Single-cell RNA sequencing data processing using cloud-based serverless computing
Curation statements for this article:-
Curated by GigaByte
Editors Assessment:
This Technical Release (software) presents a serverless cloud workflow for the computationally intensive alignment step of single-cell RNA-seq processing, extending the authors’ earlier serverless approach for bulk RNA-seq. Hung et al. built on the Piscem-Alevin-Fry pipeline and implemented it with AWS Lambda functions, working around their memory, storage and runtime limits by splitting large FASTQ files into four-million-read shards, streaming inputs from S3 straight into the aligner, and merging shard outputs as they complete. Because every stage overlaps, alignment adds only seconds beyond the cost of splitting and uploading. Compared with an optimized 32-thread on-server run on identical EC2 hardware, the serverless workflow was faster for all three datasets tested (two 10x Genomics PBMC datasets and a 450 GB, 13-sample MorPhiC pancreatic differentiation knockout dataset). The speedup grew with dataset size, from 1.27× to 1.93× to 7.01× for the largest, cutting about 2.4 hours of alignment to under 21 minutes, with outputs validated as identical to the conventional pipeline. The authors are transparent that this speed comes at a cost: the serverless runs were more expensive at list prices, though the premium shrank to about 13% for the largest dataset, and gains for small datasets are limited by driver-side preprocessing. With open-source code under an MIT license, an automated orchestration script, and deposited reference index and count matrices, this is a useful and reproducible template for processing large single-cell datasets, and for applying serverless computing to other bioinformatics workflows.
This evaluation refers to version 1 of the preprint
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (GigaByte)
Abstract
Single-cell RNA sequencing (scRNA-seq) has become a routine method for measuring cell activities. We present a novel and generalizable methodology using serverless cloud computing to accelerate computationally intensive workflows. We create an on-demand “supercomputer” using rapidly deployable cloud serverless functions as automatically provisioned computation units. We tested our methodology of optimizing an scRNA-seq workflow by leveraging serverless functions on the cloud using two publicly available peripheral blood mononuclear cell (PBMC) datasets. In addition, we demonstrate our approach using a 450 GB human scRNA-seq knockout dataset, comprising 13 samples from different developmental time points, designed to study the temporal impact of perturbations on pancreatic differentiation. We compared the execution time of the scRNA-seq serverless workflow with an optimized workflow without serverless functions running on identical hardware, and demonstrate speedups for all tested datasets, reaching 7.0-fold for the largest dataset. Our software is open source and distributed under the MIT license. Code and documentation are publicly available at https://github.com/BioDepot/scRNA-serverless.
Article activity feed
-
Editors Assessment:
This Technical Release (software) presents a serverless cloud workflow for the computationally intensive alignment step of single-cell RNA-seq processing, extending the authors’ earlier serverless approach for bulk RNA-seq. Hung et al. built on the Piscem-Alevin-Fry pipeline and implemented it with AWS Lambda functions, working around their memory, storage and runtime limits by splitting large FASTQ files into four-million-read shards, streaming inputs from S3 straight into the aligner, and merging shard outputs as they complete. Because every stage overlaps, alignment adds only seconds beyond the cost of splitting and uploading. Compared with an optimized 32-thread on-server run on identical EC2 hardware, the serverless workflow was faster for all three datasets tested (two 10x Genomics PBMC datasets and a 450 GB, …
Editors Assessment:
This Technical Release (software) presents a serverless cloud workflow for the computationally intensive alignment step of single-cell RNA-seq processing, extending the authors’ earlier serverless approach for bulk RNA-seq. Hung et al. built on the Piscem-Alevin-Fry pipeline and implemented it with AWS Lambda functions, working around their memory, storage and runtime limits by splitting large FASTQ files into four-million-read shards, streaming inputs from S3 straight into the aligner, and merging shard outputs as they complete. Because every stage overlaps, alignment adds only seconds beyond the cost of splitting and uploading. Compared with an optimized 32-thread on-server run on identical EC2 hardware, the serverless workflow was faster for all three datasets tested (two 10x Genomics PBMC datasets and a 450 GB, 13-sample MorPhiC pancreatic differentiation knockout dataset). The speedup grew with dataset size, from 1.27× to 1.93× to 7.01× for the largest, cutting about 2.4 hours of alignment to under 21 minutes, with outputs validated as identical to the conventional pipeline. The authors are transparent that this speed comes at a cost: the serverless runs were more expensive at list prices, though the premium shrank to about 13% for the largest dataset, and gains for small datasets are limited by driver-side preprocessing. With open-source code under an MIT license, an automated orchestration script, and deposited reference index and count matrices, this is a useful and reproducible template for processing large single-cell datasets, and for applying serverless computing to other bioinformatics workflows.
This evaluation refers to version 1 of the preprint
-
Singe cell RNA sequencing (scRNA-seq) has become a routine method for measuring cell activities. Processing large scRNA-seq datasets requires high-performance computing resources. The emergence of cloud computing allows us to leverage its on-demand capabilities without major investment in infrastructure. Serverless computing provides cost efficiency by allowing users to pay only for actual resource usage, eliminating the necessity for pre-allocated server capacities. Additionally, there is no requirement to set up servers in advance. We present a novel and generalizable methodology using serverless cloud computing to accelerate computationally intensive workflows. We create an on-demand “supercomputer” using rapidly deployable cloud serverless functions as automatically provisioned computation units. We tested our methodology of …
Singe cell RNA sequencing (scRNA-seq) has become a routine method for measuring cell activities. Processing large scRNA-seq datasets requires high-performance computing resources. The emergence of cloud computing allows us to leverage its on-demand capabilities without major investment in infrastructure. Serverless computing provides cost efficiency by allowing users to pay only for actual resource usage, eliminating the necessity for pre-allocated server capacities. Additionally, there is no requirement to set up servers in advance. We present a novel and generalizable methodology using serverless cloud computing to accelerate computationally intensive workflows. We create an on-demand “supercomputer” using rapidly deployable cloud serverless functions as automatically provisioned computation units. We tested our methodology of optimizing a scRNA-seq workflow by leveraging serverless functions on the cloud using two publicly available peripheral blood mononuclear cell (PBMC) datasets. In addition, we demonstrate our approach using data generated by the NIH MorPhiC program, where we process a 450 GB human scRNA-seq dataset across 86 cell lines designed to study the temporal impact of perturbations on pancreatic differentiation. We compared the total execution time of the scRNA-seq serverless workflow with the traditional workflow without using serverless functions, and demonstrate major speedup for large scRNA-seq datasets.
Reviewer 1. Boya Ji
The manuscript addresses an important and practical problem in scalable scRNA-seq data processing. I recommend acceptance after revision： 1，The current comparison uses different hardware settings for the serverless and on-server workflows, which weakens the fairness of the speedup claim. The authors should add a controlled benchmark using the same input, same Piscem-Alevin Fry parameters, same storage type, and clearly matched CPU/thread resources. 2，Since the main advantage of serverless computing is elastic, pay-as-you-go execution, the manuscript should include a cost breakdown for EC2, Lambda, S3 storage, data transfer, and ECR usage. A “cost per GB” or “cost per million reads” metric would make the practical value much clearer. 3，The paper states that count matrices are identical in constrained runs, but it should also compare serverless and non-serverless outputs using cell barcode overlap, gene count correlation, UMI totals, detected genes per cell, and downstream clustering consistency. This would show that acceleration does not alter biological results. 4，The manuscript would benefit from a sensitivity analysis of shard size, Lambda memory, thread number, concurrency limit, S3 transfer time, and merge overhead. This would help readers understand when serverless processing is beneficial and when I/O or orchestration overhead becomes the bottleneck. 5，The Related Work section should cite recent computational biology and multimodal/graph-based biomedical modeling studies to better position the work within modern bioinformatics computing: 10.1016/j.patcog.2025.112078； 10.1038/s41467-026-72882-y； 10.1093/nsr/nwaf495； 10.1186/s13059-025-03606-6
Reviewer 2. Yang Zhang
This study developed an AWS Lambda–based serverless workflow for single-cell RNA sequencing data processing. The computationally intensive read-mapping step of the Piscem–Alevin-Fry pipeline was distributed across multiple Lambda instances, while Amazon S3 was used for data transfer and result aggregation. The workflow was evaluated using two PBMC datasets and a MorPhiC knockout dataset of approximately 453 GB. The results showed that the serverless workflow achieved up to a 6.45-fold speedup for large datasets, whereas it was slower than conventional server-based processing for small datasets because of cloud resource initialization and data transfer overhead. The authors also implemented automatic AWS quota detection and resource fallback mechanisms and demonstrated that identical expression matrices could still be generated under constrained account settings. Overall, the work has practical engineering value, although the fairness of the performance comparison, cost-effectiveness, and validation of output accuracy require further improvement. Specific Comments
- The serverless workflow was orchestrated using an AWS m6id.16xlarge instance with 64 vCPUs, 256 GB RAM, and high-speed NVMe storage, whereas the conventional workflow was run on a local server equipped with an Intel i9-13900KF processor and 125 GB RAM. The two environments differ substantially in CPU architecture, core count, storage, networking, and overall computational capacity. Therefore, the reported speedup cannot be attributed solely to the serverless parallelization strategy. The authors are encouraged to add a comparison under the same cloud environment and a comparable computational resource budget, or at least discuss and normalize the impact of these hardware differences.
- The manuscript repeatedly emphasizes the pay-as-you-go model and potential cost advantages of serverless computing, but only execution time is reported. No actual costs are provided for AWS Lambda, EC2, S3 storage, data transfer, or ECR. For practical users, financial cost is as important as runtime. The authors should report the total cost of processing each dataset and compare it with the cost of conventional EC2-based or dedicated-server processing.
- The authors showed that identical count matrices were generated under different AWS quota settings, which mainly demonstrates reproducibility of the same workflow under different resource configurations. A more rigorous comparison with the standard, non-sharded Piscem–Alevin-Fry workflow is needed. The authors should compare the resulting gene expression matrices, detected cell numbers, total UMI counts, numbers of detected genes, and preferably downstream clustering results to demonstrate that FASTQ splitting and RAD file merging do not introduce computational bias.
- The reported runtimes appear to be based on single executions, without means, standard deviations, or confidence intervals. Serverless performance can vary because of Lambda cold starts, concurrent scheduling, network transfer, and fluctuations in cloud platform load. Each experiment should be repeated multiple times, and the authors should report runtime distributions, variability, and, where possible, the contribution of cold-start latency.
- The PBMC 1K dataset of 4.69 GB was processed more slowly using the serverless workflow, whereas the 44.05 GB and 453.43 GB datasets showed clear speedups. The authors should evaluate additional datasets or controlled subsets of different sizes to estimate the point at which serverless processing becomes more efficient than conventional computing. The effects of shard size, Lambda concurrency, data transfer time, and output-merging overhead should also be systematically analyzed. In addition, the phrase “Singe cell” in the title and throughout the manuscript should be corrected to “Single-cell.”
-
-
