CWL-Airflow: a lightweight pipeline manager supporting Common Workflow Language
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (GigaScience)
Abstract
Background
Massive growth in the amount of research data and computational analysis has led to increased use of pipeline managers in biomedical computational research. However, each of the >100 such managers uses its own way to describe pipelines, leading to difficulty porting workflows to different environments and therefore poor reproducibility of computational studies. For this reason, the Common Workflow Language (CWL) was recently introduced as a specification for platform-independent workflow description, and work began to transition existing pipelines and workflow managers to CWL.
Findings
Herein, we present CWL-Airflow, a package that adds support for CWL to the Apache Airflow pipeline manager. CWL-Airflow uses CWL version 1.0 specification and can run workflows on stand-alone MacOS/Linux servers, on clusters, or on a variety of cloud platforms. A sample CWL pipeline for processing of chromatin immunoprecipitation sequencing data is provided.
Conclusions
CWL-Airflow will provide users with the features of a fully fledged pipeline manager and the ability to execute CWL workflows anywhere Airflow can run—from a laptop to a cluster or cloud environment. CWL-Airflow is available under Apache License, version 2.0 (Apache-2.0), and can be downloaded from https://barski-lab.github.io/cwl-airflow, https://scicrunch.org/resolver/RRID:SCR_017196.
Article activity feed
-
Now published in GigaScience doi: 10.1093/gigascience/giz084
Michael Kotliar 1Division of Allergy and Immunology, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OHFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Michael KotliarAndrey V. Kartashov 1Division of Allergy and Immunology, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OHFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Andrey V. KartashovArtem Barski 1Division of Allergy and Immunology, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of …
Now published in GigaScience doi: 10.1093/gigascience/giz084
Michael Kotliar 1Division of Allergy and Immunology, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OHFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Michael KotliarAndrey V. Kartashov 1Division of Allergy and Immunology, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OHFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Andrey V. KartashovArtem Barski 1Division of Allergy and Immunology, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OH2Division of Human Genetics, Cincinnati Children’s Hospital Medical Center and Department of Pediatrics, College of Medicine, University of Cincinnati, Cincinnati, OHFind this author on Google ScholarFind this author on PubMedSearch for this author on this siteORCID record for Artem BarskiFor correspondence: Artem.Barski@cchmc.org
A version of this preprint has been published in the Open Access journal GigaScience (see paper https://doi.org/10.1093/gigascience/giz084 ), where the paper and peer reviews are published openly under a CC-BY 4.0 license.
These peer reviews were as follows:
Reviewer 1: http://dx.doi.org/10.5524/REVIEW.101838 Reviewer 2: http://dx.doi.org/10.5524/REVIEW.101839
-
-
-