Hot-starting software containers for STAR aligner
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (GigaScience)
Abstract
Background
Using software containers has become standard practice to reproducibly deploy and execute biomedical workflows on the cloud. However, some applications that contain time-consuming initialization steps will produce unnecessary costs for repeated executions.
Findings
We demonstrate that hot-starting from containers that have been frozen after the application has already begun execution can speed up bioinformatics workflows by avoiding repetitive initialization steps. We use an open-source tool called Checkpoint and Restore in Userspace (CRIU) to save the state of the containers as a collection of checkpoint files on disk after it has read in the indices. The resulting checkpoint files are migrated to the host, and CRIU is used to regenerate the containers in that ready-to-run hot-start state. As a proof-of-concept example, we create a hot-start container for the spliced transcripts alignment to a reference (STAR) aligner and deploy this container to align RNA sequencing data. We compare the performance of the alignment step with and without checkpoints on cloud platforms using local and network disks.
Conclusions
We demonstrate that hot-starting Docker containers from snapshots taken after repetitive initialization steps are completed significantly speeds up the execution of the STAR aligner on all experimental platforms, including Amazon Web Services, Microsoft Azure, and local virtual machines. Our method can be potentially employed in other bioinformatics applications in which a checkpoint can be inserted after a repetitive initialization phase.
Article activity feed
-
Now published in GigaScience doi: 10.1093/gigascience/giy092
Pai Zhang Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteLing-Hong Hung Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteWes Lloyd Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteKa Yee Yeung Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this site
A version of this preprint has been …
Now published in GigaScience doi: 10.1093/gigascience/giy092
Pai Zhang Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteLing-Hong Hung Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteWes Lloyd Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteKa Yee Yeung Institute of Technology, University of Washington, Tacoma, Washington 98402, USAFind this author on Google ScholarFind this author on PubMedSearch for this author on this site
A version of this preprint has been published in the Open Access journal GigaScience (see paper https://doi.org/10.1093/gigascience/giy092 ), where the paper and peer reviews are published openly under a CC-BY 4.0 license.
These peer reviews were as follows:
Reviewer 1: http://dx.doi.org/10.5524/REVIEW.101280 Reviewer 2: http://dx.doi.org/10.5524/REVIEW.101281 Reviewer 3: http://dx.doi.org/10.5524/REVIEW.101282
-
-
