Accumulating computational resource usage of genomic data analysis workflow to optimize cloud computing instance selection
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (GigaScience)
Abstract
Background
Container virtualization technologies such as Docker are popular in the bioinformatics domain because they improve the portability and reproducibility of software deployment. Along with software packaged in containers, the standardized workflow descriptors Common Workflow Language (CWL) enable data to be easily analyzed on multiple computing environments. These technologies accelerate the use of on-demand cloud computing platforms, which can be scaled according to the quantity of data. However, to optimize the time and budgetary restraints of cloud usage, users must select a suitable instance type that corresponds to the resource requirements of their workflows.
Results
We developed CWL-metrics, a utility tool for cwltool (the reference implementation of CWL), to collect runtime metrics of Docker containers and workflow metadata to analyze workflow resource requirements. To demonstrate the use of this tool, we analyzed 7 transcriptome quantification workflows on 6 instance types. The results revealed that choice of instance type can deliver lower financial costs and faster execution times using the required amount of computational resources.
Conclusions
CWL-metrics can generate a summary of resource requirements for workflow executions, which can help users to optimize their use of cloud computing by selecting appropriate instances. The runtime metrics data generated by CWL-metrics can also help users to share workflows between different workflow management frameworks.
Article activity feed
-
Now published in GigaScience doi: 10.1093/gigascience/giz052
Tazro Ohta 1Database Center for Life Science, Joint Support-Center for Data Science Research, Research Organization of Information and Systems, Yata 1111, Mishima, Shizuoka 411-8540, JapanFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Tazro OhtaTomoya Tanjo 2National Institute of Informatics, Research Organization of Information and Systems, Tokyo 101–8430, JapanFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Tomoya TanjoOsamu Ogasawara 3DNA Data Bank of Japan, National Institute of Genetics, Research Organization of Information and Systems, Yata, Mishima 411-8540, JapanFind this author on Google ScholarFind this author on PubMedSearch for this …
Now published in GigaScience doi: 10.1093/gigascience/giz052
Tazro Ohta 1Database Center for Life Science, Joint Support-Center for Data Science Research, Research Organization of Information and Systems, Yata 1111, Mishima, Shizuoka 411-8540, JapanFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Tazro OhtaTomoya Tanjo 2National Institute of Informatics, Research Organization of Information and Systems, Tokyo 101–8430, JapanFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Tomoya TanjoOsamu Ogasawara 3DNA Data Bank of Japan, National Institute of Genetics, Research Organization of Information and Systems, Yata, Mishima 411-8540, JapanFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Osamu Ogasawara
A version of this preprint has been published in the Open Access journal GigaScience (see paper https://doi.org/10.1093/gigascience/giz052 ), where the paper and peer reviews are published openly under a CC-BY 4.0 license.
These peer reviews were as follows:
Reviewer 1: http://dx.doi.org/10.5524/REVIEW.101638 Reviewer 2: http://dx.doi.org/10.5524/REVIEW.101639 Reviewer 3: http://dx.doi.org/10.5524/REVIEW.101640
-
-
