Systematic comparison of ranking aggregation methods for gene lists in experimental results
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Motivation
A common experimental output in biomedical science is a list of genes implicated in a given biological process or disease. The gene lists resulting from a group of studies answering the same, or similar, questions can be combined by ranking aggregation methods to find a consensus or a more reliable answer. Evaluating a ranking aggregation method on a specific type of data before using it is required to support the reliability since the property of a dataset can influence the performance of an algorithm. Such evaluation on gene lists is usually based on a simulated database because of the lack of a known truth for real data. However, simulated datasets tend to be too small compared to experimental data and neglect key features, including heterogeneity of quality, relevance and the inclusion of unranked lists.
Results
In this study, a group of existing methods and their variations that are suitable for meta-analysis of gene lists are compared using simulated and real data. Simulated data were used to explore the performance of the aggregation methods as a function of emulating the common scenarios of real genomic data, with various heterogeneity of quality, noise level and a mix of unranked and ranked data using 20 000 possible entities. In addition to the evaluation with simulated data, a comparison using real genomic data on the SARS-CoV-2 virus, cancer (non-small cell lung cancer) and bacteria (macrophage apoptosis) was performed. We summarize the results of our evaluation in a simple flowchart to select a ranking aggregation method, and in an automated implementation using the meta-analysis by information content algorithm to infer heterogeneity of data quality across input datasets.
Availability and implementation
The code for simulated data generation and running edited version of algorithms: https://github.com/baillielab/comparison_of_RA_methods. Code to perform an optimal selection of methods based on the results of this review, using the MAIC algorithm to infer the characteristics of an input dataset, can be downloaded here: https://github.com/baillielab/maic. An online service for running MAIC: https://baillielab.net/maic.
Supplementary information
Supplementary data are available at Bioinformatics online.
Article activity feed
-
-
SciScore for 10.1101/2022.01.09.475491: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Software and Algorithms Sentences Resources The implementation language for MAIC, VC, and RepeatChoice is Python whereas it is R for all other investigated methods in Table 4 except for BARD, which has available code implemented in C++. Pythonsuggested: (IPython, RRID:SCR_001658)Similar to the research by Li et al. (2019) [1], Borda’s methods are labelled with ‘r’ and ‘t’ to show different implementations from the RobustRankAggreg package and TopKList package. TopKListsuggested: NoneBARD [14] defines a model with an independent parameter for each list to control the probability of position for true entities among noise entities … SciScore for 10.1101/2022.01.09.475491: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Software and Algorithms Sentences Resources The implementation language for MAIC, VC, and RepeatChoice is Python whereas it is R for all other investigated methods in Table 4 except for BARD, which has available code implemented in C++. Pythonsuggested: (IPython, RRID:SCR_001658)Similar to the research by Li et al. (2019) [1], Borda’s methods are labelled with ‘r’ and ‘t’ to show different implementations from the RobustRankAggreg package and TopKList package. TopKListsuggested: NoneBARD [14] defines a model with an independent parameter for each list to control the probability of position for true entities among noise entities within the corresponding list. BARDsuggested: (BARD, RRID:SCR_006283)Results from OddPub: Thank you for sharing your code and data.
Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
Results from scite Reference Check: We found no unreliable references.
-
