Signature-scoring methods developed for bulk samples are not adequate for cancer single-cell RNA sequencing data
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
Single-cell RNA sequencing allows us to quantify variability of gene expression patterns within a population, and thus devise patterns with prognostic, diagnostic and predictive potential, called "gene expression signatures". Here, Noureen and collaborators benchmark four methods used for identifying these gene expression signatures, and evaluate their performance at overcoming a number of analytical challenges. They conclude that caution should be exercised when using bulk sample-based methods in single-cell data analyses, and cellular contexts should be taken into consideration. With strengthening of some statistical and methodological aspects to support the validity of the conclusions, this paper will be an informative and potentially valuable addition to the literature.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #3 agreed to share their name with the authors.)
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Computational and Systems Biology (eLife)
Abstract
Quantifying the activity of gene expression signatures is common in analyses of single-cell RNA sequencing data. Methods originally developed for bulk samples are often used for this purpose without accounting for contextual differences between bulk and single-cell data. More broadly, few attempts have been made to benchmark these methods. Here, we benchmark five such methods, including single sample gene set enrichment analysis (ssGSEA), Gene Set Variation Analysis (GSVA), AUCell, Single Cell Signature Explorer (SCSE), and a new method we developed, Jointly Assessing Signature Mean and Inferring Enrichment (JASMINE). Using cancer as an example, we show cancer cells consistently express more genes than normal cells. This imbalance leads to bias in performance by bulk-sample-based ssGSEA in gold standard tests and down sampling experiments. In contrast, single-cell-based methods are less susceptible. Our results suggest caution should be exercised when using bulk-sample-based methods in single-cell data analyses, and cellular contexts should be taken into consideration when designing benchmarking strategies.
Article activity feed
-
-
Evaluation Summary:
Single-cell RNA sequencing allows us to quantify variability of gene expression patterns within a population, and thus devise patterns with prognostic, diagnostic and predictive potential, called "gene expression signatures". Here, Noureen and collaborators benchmark four methods used for identifying these gene expression signatures, and evaluate their performance at overcoming a number of analytical challenges. They conclude that caution should be exercised when using bulk sample-based methods in single-cell data analyses, and cellular contexts should be taken into consideration. With strengthening of some statistical and methodological aspects to support the validity of the conclusions, this paper will be an informative and potentially valuable addition to the literature.
(This preprint has been reviewed by eLife. …
Evaluation Summary:
Single-cell RNA sequencing allows us to quantify variability of gene expression patterns within a population, and thus devise patterns with prognostic, diagnostic and predictive potential, called "gene expression signatures". Here, Noureen and collaborators benchmark four methods used for identifying these gene expression signatures, and evaluate their performance at overcoming a number of analytical challenges. They conclude that caution should be exercised when using bulk sample-based methods in single-cell data analyses, and cellular contexts should be taken into consideration. With strengthening of some statistical and methodological aspects to support the validity of the conclusions, this paper will be an informative and potentially valuable addition to the literature.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #3 agreed to share their name with the authors.)
-
Reviewer #1 (Public Review):
In this study the authors attempted to benchmark different methods for gene signature enrichment methods in single-cell RNA-seq data. They compared 2 single-cell based methods (AUCell, SCSE), their own developed method (JASMINE) and a popular method used in bulk RNA-seq studies (ssGSEA). For the benchmarking they collected 10 studies of scRNA-seq data from tumors, and performed several statistical analysis. In all the analyses ssGSEA performed worse by having lower specificity.
In this reviewer opinion, this is an important and understudied subject. In many scRNA-seq studies there is use of gene signatures to make a point, but to my knowledge there haven't been any deep-dive assessment of the performance of the methodologies used. Specifically, many studies use ssGSEA, but there haven't been any assessment …
Reviewer #1 (Public Review):
In this study the authors attempted to benchmark different methods for gene signature enrichment methods in single-cell RNA-seq data. They compared 2 single-cell based methods (AUCell, SCSE), their own developed method (JASMINE) and a popular method used in bulk RNA-seq studies (ssGSEA). For the benchmarking they collected 10 studies of scRNA-seq data from tumors, and performed several statistical analysis. In all the analyses ssGSEA performed worse by having lower specificity.
In this reviewer opinion, this is an important and understudied subject. In many scRNA-seq studies there is use of gene signatures to make a point, but to my knowledge there haven't been any deep-dive assessment of the performance of the methodologies used. Specifically, many studies use ssGSEA, but there haven't been any assessment of its reliability in single cell studies. As the authors note, gene dropouts in scRNA-seq may have major effect on reliability of this method.
My problem with this study is that the benchmarking is focused only on comparison of statistical measurements, but in my opinion, this is problematic here. The aim of an enrichment method is to identify biological relevant pathways. There is no analysis here that looks at the relevance of the significant signatures.
Here is an example of the main weakness - the main analysis performed by the authors was identifying gene sets that significantly distinguish between cancer cells and the rest of the cells in the tumor. This a problematic comparison, since the cancer cells are epithelial cells (in most of the studies used) and the "normal" cells are stromal cells, mostly immune. Those are not comparable "normal" cells, and therefore it is expected that all immune-related pathways will be significant. The authors find much more down-regulated gene sets in ssGSEA compared to the other methods, but why are they wrong? If they are all immune related, I would actually conclude that ssGSEA is better than the other methods.
Another weakness in this study is that the authors relate to the methods as if they are black boxes. If the main results of this study is that bulk gene expression methods (only one method is assessed, so I don't understand the title) cannot be used in single-cell data, I would expect to learn from the study what in the methodology makes it problematic.
The bottom line is that the only conclusion I can deduce from this study is that ssGSEA provides different results from newer single-cell specific methods.
-
Reviewer #2 (Public Review):
Noureen et al. benchmark four methods for quantifying the activity of gene expression signatures in single cell data, including one they developed, called JASMINE. They point to an imbalance in the number of expressed genes between tumor and normal cells, which they claim leads to a bias in performance of such methods.
The authors emphasize an important message -- considering cellular context when analyzing differentially expressed genes and gene signatures. Another strength is that the datasets on which these methods were evaluated is relatively large. However, they evaluate only four signature-scoring methods - a major weakness of the study. The new method they propose includes formulations which lead to unintended mathematical behavior and hamper interpretation.
-
Reviewer #3 (Public Review):
In this manuscript, Noureen et al. benchmarked four methods for the gene expression signature analysis for single-cell RNA sequencing data. They showed that cancer cells consistently expressing more genes than normal cells is the major factor to cause the bias. They also developed a method, JASMINE, and benchmarked this method with other methods. They finally suggest that cellular contexts should be taken into consideration for single-cell data analysis.
The topic of the study is important, considering a large amount of single-cell data released recently. The manuscript is well-organized and well-written. The strength is that the manuscript provides clear guidance for future benchmarking of the single-cell data analysis.
-
-