CytofRUV: Removing unwanted variation to integrate multiple CyTOF datasets

  1. Marie Trussart
  2. Charis E Teh
  3. Tania Tan
  4. Lawrence Leong
  5. Daniel HD Gray
  6. Terence P. Speed

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

Mass cytometry (CyTOF) is a technology that has revolutionised single cell biology. One illuminating application of CyTOF has been in understanding the mechanisms of blood cancer resistance to therapy. Longitudinal studies of clinical cohorts during drug treatment provide a deeper understanding of the molecular changes that underlie sensitivity or resistance to treatment in each patient. However, understanding the biological impact of a cancer drug in such studies necessitates the integration of multiple CyTOF batches. To date, the integration of CyTOF datasets remains a challenge due to technical differences arising in multiple batches. To overcome this limitation, we developed an approach called CytofRUV for analysing multiple CyTOF batches which includes an R-Shiny application with diagnostics plots. CytofRUV can correct for batch effects and integrate data from large numbers of patients and conditions across batches, to confidently compare cellular changes and correlate these with clinically relevant outcomes.

Article activity feed

  1. eLife

    ###Reviewer #4:

    This paper presents CytofRUV, a new tool to remove technical batch effects in CYTOF data, inspired by tools used in the transcriptomics field. There is still a strong need for such tools and I expect this tool to be a valuable addition to the cytometry field. I especially appreciate the authors' effort in providing multiple evaluation measures and informative figures to estimate the properties of the batch effects before and after normalization. There is currently no one-fits-all solution for batch normalization, and having sufficient quality control along the way is absolutely invaluable.

    I recommend no major changes to the manuscript, but mainly some additional guidance in the reader's interpretation of some results, and some smaller suggestions to improve figures. Some of the more unexpected results are not commented on in the text and it would be helpful if some interpretation could be given in those cases.

    -Many methods cause an increased batch silhouette score compared to raw, does this mean that in those cases the methods increase the batch effects?

    -Also the Hellinger distances sometimes become bigger than originally. Would there be any way to check if this distance would be small given an adapted manual gating? Or could there be any reason that actually some cell types are indeed differing in proportion in the different batches, so you would not expect the batch correction to "restore" this (as no cells are added or removed by the correction)? As both CytoNorm and CytofRUV apply the normalization on a cluster-by-cluster basis, I am also not sure why the cluster proportions afterwards would become more similar. Can you give any further intuition about this?

    -While there is a section regarding "keeping biological differences" this is only explored on the population level in the individual samples. I would also find it of interest to read something about biological differences between samples which are preserved (e.g. maybe quantifying the differences between the healthy controls?)

  2. eLife

    ###Reviewer #3:

    The manuscript in review discusses a new method to address technical variances in CYTOF data called CytofRUV and based on Remove Unwanted Variation methodology. CYTOF datasets are prone to significant batch-to-batch variation due to the technical nature of signal registration and this method adds to the group of previously published algorithms aimed to solve the same task.

    The manuscript is well-written and the narrative flows well. The authors come up with compelling examples of batch effect in CYTOF data (e.g. Fig.2) that honestly not only call for robust algorithmic normalization but make me somewhat question the claimed reliability of the CYTOF technology to deliver precise measurement of protein expression without robust replicates built into every experimental design of CYTOF experiments; this publication would surely raise awareness of existing issues. Authors also line up a series of metrics to quantify the efficiency of theirs and alternative methods for data normalization, and propose a strong battery of visual cues built into their Shiny app to evaluate the algorithm results.

    1. The algorithm performance deserves more discussion that is currently outsourced to the reference to original RUV paper (Molania et al). How computationally demanding is it? What computational resources were used? How does it scale to large datasets? How parametrization (choice of k value) affects the results specifically for CYTOF data (this is slightly touched upon in the Molania et al paper, but the data context is very different)?

    2. Are any of the metrics mentioned in the paper built into the R package/Shiny app? From the paper, it looks like the only outputs that the interface presents are the four visual plots but no evaluation metrics of how the normalization affected/improved the data.

    3. Besides silhouette scores, were there any other attempts to verify the data integrity post processing? For instance, how reproducible are clustering results after normalization if the processed data are clustered from scratch and compared to clustering performed before normalization?

    4. Based on existing datasets and metric outputs, would the authors suggest a way to estimate the minimal number of replicates (as discussed in lines 488-492) required for the specific panel/sample/instrument type to provide necessary power to preserve the resolution of the data post normalization?

  3. eLife

    ###Reviewer #2:

    The authors have presented a novel approach based on RUV-III for normalizing CyTOF data leveraging replicate samples across batches. The article is clear, well laid out, thoughtful and presents well-substantiated conclusions. The RUV class of method has been applied across high throughput technologies including RNASeq, single-cell RNASeq, nanostring and others and it is a natural extension to single cell cytometry. I have few issues with the paper. My one minor concern is the conflation of the term cell subpopulation with cluster. I don't think this detracts from the conclusions of the paper, but the former typically is reserved for cells of a consistent and verified phenotype. FlowSOM and just about all other clustering methods do not necessarily produce clusters that correspond to consistent cell sub populations (the phenotype of the cells included in a cluster can and does vary). I think to make statements about sub populations, the authors would have to look at manual phenotype assignments as well. I am not suggesting that it is necessary, and I find the evaluation of the method with respect to clusters much more compelling and natural. However, I would request that the authors make the distinction between clusters and cell sub populations in this context.

    After looking at the software implementation I think some discussion of the computational complexity and limitations of the method and implementation is warranted, particularly time and memory considerations. Could the method scale to large data sets (100s or 1000s of samples with several 100k cells each), which are typical in clinical studies? Do all data need to be loaded into working memory for the current implementation, or in general?

  4. eLife

    ###Reviewer #1:

    The article describes CytofRUV, an algorithm for normalization of mass cytometry datasets. The article is well written, the data is publicly available, and the source code is usable and well-documented. My comments are provided below:

    Major comments:

    1. I believe the focus of this article can be improved. The abstract is a bit confusing. If the article is focused on the algorithm, the focus of the abstract should not be on leukemia. This can be used in many settings. Similarly, much of the article (including 4 of the main figures) are dedicated to establishing that this one dataset indeed does have a batch effect issue. Other datasets are not introduced until the very end of the manuscript. However, for an article focused on the development of a new bioinformatics method, I believe the focus should be on evaluation of the algorithm on a broad range of datasets (which the authors have already done, but should be presented more prominently).

    2. Comparison with prior algorithms is only presented in a qualitative manner. Quantification of these comparisons, followed by appropriate statistical tests, would strengthen this article. I don't believe a new algorithm needs to outperform existing algorithms in every test (as it runs against the no free lunch theorem) but quantification should be provided regardless.

  5. eLife

    ##Preprint Review

    This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 2 of the manuscript.

    ###Summary:

    The authors present a new Cytof normalization approach based on RUV III that has proven useful for other technologies including RNASeq, single-cell RNAseq and nanostring. The reviewers all agreed that this was a strong manuscript that makes an important contribution to an area of the field that remains under-served.

  6. Version published to 10.1101/2020.05.09.085621 on bioRxiv