Unsupervised machine learning reveals key immune cell subsets in COVID-19, rhinovirus infection, and cancer therapy

  1. Sierra M Barone
  2. Alberta GA Paul
  3. Lyndsey M Muehling
  4. Joanne A Lannigan
  5. William W Kwok
  6. Ronald B Turner
  7. Judith A Woodfolk
  8. Jonathan M Irish

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    The three reviewers were overly positive about the stated goal of your T-REX method to characterize rare populations by cytometry. The potential applications in the context of analyzing antigen-specific T cells (as identified as tetramer-positive cells) were not missed on the reviewers and the use of the 2 timepoints-cohort of samples from rhinovirus-infected patients was judged clever. However, all three reviewers requested some edits and additional tests to really distinguish T-REX from other methods in terms of performance, and to better understand its analysis power. Reviewer #1 enjoined you to clarify the improvements of your method compared to previous methods. Reviewer #2 requested more stringent tests of your methods against functional datasets. Reviewer #3 inquired about corrections for batch effects, the result consistency for repeated down-sampling as well as the scalability of the method (especially when UMAP is being used).

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 and Reviewer #3 agreed to share their names with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

For an emerging disease like COVID-19, systems immunology tools may quickly identify and quantitatively characterize cells associated with disease progression or clinical response. With repeated sampling, immune monitoring creates a real-time portrait of the cells reacting to a novel virus before disease-specific knowledge and tools are established. However, single cell analysis tools can struggle to reveal rare cells that are under 0.1% of the population. Here, the machine learning workflow Tracking Responders EXpanding (T-REX) was created to identify changes in both rare and common cells across human immune monitoring settings. T-REX identified cells with highly similar phenotypes that localized to hotspots of significant change during rhinovirus and SARS-CoV-2 infections. Specialized MHCII tetramer reagents that mark rhinovirus-specific CD4+ cells were left out during analysis and then used to test whether T-REX identified biologically significant cells. T-REX identified rhinovirus-specific CD4+ T cells based on phenotypically homogeneous cells expanding by ≥95% following infection. T-REX successfully identified hotspots of virus-specific T cells by comparing infection (day 7) to either pre-infection (day 0) or post-infection (day 28) samples. Plotting the direction and degree of change for each individual donor provided a useful summary view and revealed patterns of immune system behavior across immune monitoring settings. For example, the magnitude and direction of change in some COVID-19 patients was comparable to blast crisis acute myeloid leukemia patients undergoing a complete response to chemotherapy. Other COVID-19 patients instead displayed an immune trajectory like that seen in rhinovirus infection or checkpoint inhibitor therapy for melanoma. The T-REX algorithm thus rapidly identifies and characterizes mechanistically significant cells and places emerging diseases into a systems immunology context for comparison to well-studied immune changes.

Article activity feed

  1. Version published to 10.7554/elife.64653 on eLife
  2. eLife

    Author Response:

    Reviewer #1:

    Sierra M. Barone developed an automated, quantitative toolkit for immune monitoring that would span a wide range of possible immune changes, identify and phenotype statistically significant cell subsets, and provide an overall vector of change indicating both the direction and magnitude of shifts, either in the immune system as a whole or in a key cell subpopulation. The machine learning workflow Tracking Responders Expanding (T-REX) was a modular data analysis workflow including UMAP, KNN, and MEM. T-REX is designed to capture both very rare and very common cell types and place them into a common context of immune change. T-REX was analyzed data types including a new spectral flow cytometry dataset and three existing mass cytometry datasets.

    The conclusions of this paper are mostly well supported by data, but one aspect need to be clarified and extend. Cytometry tools like SPADE, FlowSOM, Phenograph, Citrus, and RAPID generally work best to characterize cell subsets representing >1% of the sample and are less capable of capturing extremely rare cells or subsets distinguished by only a fraction of measured features. Tools like t-SNE, opt-SNE, and UMAP embed cells or learn a manifold and represent these transformations as algorithmically-generated axes.

    We appreciate this point and have added a new Figure 7 that quantifies performance vs. commonly used analysis tools.

    The advantages of T-REX tool were not very clear.

    We believe the newly added Figure 7 and associated text in the Results helps to show the advantages of T-REX (and specific choices in the algorithm) over other tools and algorithm choices. We make 9 different comparisons and use four widely-used algorithms from cytometry, including t-SNE, FlowSOM, Phenograph, Citrus, and SPADE.

    Reviewer #2:

    This study presents a novel machine learning tool (termed T-REX) for automated analysis of single cell cytometric data that is capable of identifying rare cell populations, such as antigen-specific T cells. This ability to detect low frequency cells is a distinct advantage over existing tools. The demonstration of this ability is appropriately shown by examining antigen-specific CD4+ T cells before and after rhinovirus infection in a challenge study. Useful demonstrations are also included for examining SARS-CoV-2-specific T cells and changes in cellular populations in cancer patients upon treatment. These examples use both mass cytometry and fluorescence-based cytometry. Since both of these are commonly-used single-cell technologies that generate highly complex data sets, new automated analysis methods such as T-REX are needed.

    The first data set examined changes in cell phenotype before and 7 days after rhinovirus infection in healthy adults. The flow cytometric staining panel included markers of T cell differentiation and activation as well as rhinovirus-specific tetramers. The results of T-REX convincingly demonstrate "hotspots" that are expanded at 7 days and enriched for tetramer-staining cells. Thus, this study succeeds at demonstrating the utility of this method for identification of rare cells and the authors use this data set to appropriately determine the model parameters. Combining the results of this algorithm with the "Marker Enrichment Modeling (MEM)" method to characterize the markers expressed on those cell populations identified through T-REX is also very informative since this automates the characterization (that traditionally needs to be done by manual investigation).

    This first data set is relevant for this demonstration, but in some aspects it represents a best case scenario. "Phenotypic" identification of antigen-specific T cells in this way is only possible because the time point was chosen to capture the relatively narrow window when T cells would be activated, and there was access to a baseline sample for comparison. The authors do address the second point, and perform the analysis comparing day 7 to a later time point, day 28, as an appropriate alternative. The first concern limits the generalizability of this approach. In fact, the second example dataset examining mass cytometry data in patients with COVID-19 does in fact demonstrate limited ability to detect change in cell populations for many study participants.

    We appreciate the point that the rhinovirus dataset timepoints were selected carefully to help reveal the antigen specific T cells and that in other disease settings one may not know the optimal timepoint and need to sample multiple times. We have added a note addressing this in the Discussion. To the second point, we agree that in the COVID-19 response example that it is difficult to pinpoint the antigen specific cells. We believe this is due to the high degree of overall change in the immune system and agree that it could also not represent the ideal timepoint (i.e., there are multiple reasons it is a different scenario). This is also captured in the updated Discussion text thinking through the applicability of T-REX to different scenarios. Notably, we predict that in the COVID-19 vaccine response setting, where there is far less overall immune change, that T-REX might fruitfully identify antigen specific cells (this is beyond the scope of the present manuscript, but mentioned as a next step in the Discussion).

    Reviewer #3:

    Barone, Paul et al. present a new computational method, named T-REX, to detect changes in immune cell populations from repeated cytometry measurements (before and after infection or treatment). The proposed method is designed to detect changes in rare and common cells with particular focus on the former. T-REX detects subpopulations of cells showing marked differences in abundance between the proportion of cells from different time points (before and after infection) from a single individual. The method relies of a dimensionality reduction step using UMAP followed by a K-nearest neighbor (KNN) search to identify cells that have a large fraction (>0.95) of neighbors from one time point, indicating expansion or shrinkage of certain cell populations. Areas in the UMAP with clustered expanding or shrinking neighborhoods are labeled as hotspots. Cells in these hotspots were further characterized and enriched markers were identified using MEM, a method published earlier by the same authors. T-REX was applied to a newly collected dataset of rhinovirus infection and three publicly available datasets of SARS-CoV-2 infections, melanoma immunotherapy and AML chemotherapy. The results are presented clearly and the authors discuss in details several examples in which the cells identified by T-REX have a phenotypic profile which align with previous knowledge, indicating the relevance of the results.

    Strengths:

    • T-REX is based on a simple pipeline including UMAP and KNN. This is an advantage especially given the large number of cells collected. Further, the proposed approach has a key advantage since it allows the analysis of one sample at a time, which is practical if one wants to analyze a new sample. There is no need to rerun the analysis on an aggregate of a large number of samples.
    • The new rhinovirus dataset is of great value to the community.

    Weaknesses:

    • The paper lacks a comparison to other methods for differential abundance testing. In particular, it is not clear how T-REX differs from the Differential abundance test proposed by Lun et al. (https://doi.org/10.1038/nmeth.4295). Similarly, there are no experiments or results to support the authors' initial claim that T-REX outperforms current clustering-based methods (SPADE, FLOWSOM, Phenograph,…etc.) in capturing changes in rare (<1%) cell populations.

    We appreciate this point and have selected a few of the top algorithms, as well as algorithms we and collaborators use regularly, and we made direct performance comparisons of accuracy and time to support our initial claim (new Figure 7 and associated Results paragraph).

    • T-REX relies on arbitrary cutoffs (0.95 and 0.5 %) to define expansion or shrinkage in the neighborhood of each cell (0.95 and 0.5 %) rather than a formal statistical test. These cut-offs were defined based on the ability to detect tetramer positive cells in one subject only. This greatly limits the generalizability of the method.

    We appreciate these points that the ‘optimal’ k value was determined using a single individual and that a single one size fits all cutoff may not be ideal for all situations. In the case of rhinovirus, we can use the tetramer+ cells as a type of truth, and we proceeded to use the additional patients to test for optimal k-values. The results of this analysis are referred to as numbers in the Results text and we have added text to highlight this point. Briefly, in continued analysis of the rest of the infected rhinovirus subjects, optimal k values ranged from 30 to 80. Additional optimization based on a formal statistical test is something we would like to explore in a subsequent study, but is beyond the scope of this manuscript. We have noted both points around the k-value and the cutoff in new text added to the Discussion. In other biological studies, we have seen a T-REX change cutoff of 85% be useful, so we imagine users will need to test this on their own for their biological questions.

    • The authors do not motivate the use of UMAP prior to the KNN graph reconstruction. While UMAP is a clearly powerful method to visualize single cell data, the resulting embedding can potentially show distinct groups of points when the high dimensional manifold is more continuous. For this reason, KNN graphs are usually built using the high-dimensional data (or principal components).

    We have added a new figure that includes comparing KNN on the UMAP coordinates to KNN on the original high-dimensional feature space as well as other comparisons (Figure 7). The analysis on UMAP or t-SNE axes outperformed analysis of high-dimensional data (termed “original features” in the text) when using KNN and Phenograph. SPADE was the only algorithm that identified both significant regions of change when starting with the original features.

    • Given that T-REX is mainly developed to detect changes in rare cell populations, the paper lacks an assessment of the method's sensitivity. For instance, cells were subsampled equally from each time point. An assessment of the effects of this subsampling step is necessary. In general, a guide to the users indicating the limitations of T-REX will be greatly helpful.

    We appreciate this point and have added a new Supplemental Figure 7 to assess the sensitivity of T- REX with subsampling. We have also expanded on the limitations and uses of T-REX in the discussion.

    • Given that the main aim of T-REX is to detect differences in rare cells, the rational to perform a separate analysis for CD4 positive cells is not clear. One would expect these differences to be identified also in the analysis performed using all cells.

    We appreciate this point and agree that T-REX using all the cells in the rhinovirus study could be interesting. The main focus in this study was on CD4+ T cells specifically, as rhinovirus is known to induce expansion of circulating virus-specific CD4+ T cells in the blood and these T cells were the only ones marked by tetramers in this study. We did run T-REX on all the cells from a rhinovirus subject (RV001) and still were able to capture the rare virus-specific, CD4+ T cells. We have added a new Supplemental Figure 1 with that result. However, CD4+ were the target of the analysis used to test the algorithm and compare to a known “truth” in that population of cells.

    • The paper lacks a discussion on the effects of batch effects between the different time points on the performance of T-REX.

    We appreciate this point and have addressed limitations and considerations with batch effects in the Discussion. Notably, in the rhinovirus study, each subject is from a single batch (so different batches were not pooled for T-REX, but the results of T-REX were comparable across batches, as seen by the same MEM label phenotype being revealed across subjects). Separate batches would need batch normalization before being run through T-REX, especially when using t-SNE or UMAP, since T- REX is designed to be very sensitive to slight changes.

  3. eLife

    Reviewer #3 (Public Review):

    Barone, Paul et al. present a new computational method, named T-REX, to detect changes in immune cell populations from repeated cytometry measurements (before and after infection or treatment). The proposed method is designed to detect changes in rare and common cells with particular focus on the former. T-REX detects subpopulations of cells showing marked differences in abundance between the proportion of cells from different time points (before and after infection) from a single individual. The method relies of a dimensionality reduction step using UMAP followed by a K-nearest neighbor (KNN) search to identify cells that have a large fraction (>0.95) of neighbors from one time point, indicating expansion or shrinkage of certain cell populations. Areas in the UMAP with clustered expanding or shrinking neighborhoods are labeled as hotspots. Cells in these hotspots were further characterized and enriched markers were identified using MEM, a method published earlier by the same authors. T-REX was applied to a newly collected dataset of rhinovirus infection and three publicly available datasets of SARS-CoV-2 infections, melanoma immunotherapy and AML chemotherapy. The results are presented clearly and the authors discuss in details several examples in which the cells identified by T-REX have a phenotypic profile which align with previous knowledge, indicating the relevance of the results.

    Strengths:

    • T-REX is based on a simple pipeline including UMAP and KNN. This is an advantage especially given the large number of cells collected. Further, the proposed approach has a key advantage since it allows the analysis of one sample at a time, which is practical if one wants to analyze a new sample. There is no need to rerun the analysis on an aggregate of a large number of samples.

    • The new rhinovirus dataset is of great value to the community.

    Weaknesses:

    • The paper lacks a comparison to other methods for differential abundance testing. In particular, it is not clear how T-REX differs from the Differential abundance test proposed by Lun et al. (https://doi.org/10.1038/nmeth.4295). Similarly, there are no experiments or results to support the authors' initial claim that T-REX outperforms current clustering-based methods (SPADE, FLOWSOM, Phenograph,...etc.) in capturing changes in rare (<1%) cell populations.

    • T-REX relies on arbitrary cutoffs (0.95 and 0.5 %) to define expansion or shrinkage in the neighborhood of each cell (0.95 and 0.5 %) rather than a formal statistical test. These cut-offs were defined based on the ability to detect tetramer positive cells in one subject only. This greatly limits the generalizability of the method.

    • The authors do not motivate the use of UMAP prior to the KNN graph reconstruction. While UMAP is a clearly powerful method to visualize single cell data, the resulting embedding can potentially show distinct groups of points when the high dimensional manifold is more continuous. For this reason, KNN graphs are usually built using the high-dimensional data (or principal components).

    • Given that T-REX is mainly developed to detect changes in rare cell populations, the paper lacks an assessment of the method's sensitivity. For instance, cells were subsampled equally from each time point. An assessment of the effects of this subsampling step is necessary. In general, a guide to the users indicating the limitations of T-REX will be greatly helpful.

    • Given that the main aim of T-REX is to detect differences in rare cells, the rational to perform a separate analysis for CD4 positive cells is not clear. One would expect these differences to be identified also in the analysis performed using all cells.

    • The paper lacks a discussion on the effects of batch effects between the different time points on the performance of T-REX.

  4. eLife

    Reviewer #2 (Public Review):

    This study presents a novel machine learning tool (termed T-REX) for automated analysis of single cell cytometric data that is capable of identifying rare cell populations, such as antigen-specific T cells. This ability to detect low frequency cells is a distinct advantage over existing tools. The demonstration of this ability is appropriately shown by examining antigen-specific CD4+ T cells before and after rhinovirus infection in a challenge study. Useful demonstrations are also included for examining SARS-CoV-2-specific T cells and changes in cellular populations in cancer patients upon treatment. These examples use both mass cytometry and fluorescence-based cytometry. Since both of these are commonly-used single-cell technologies that generate highly complex data sets, new automated analysis methods such as T-REX are needed.

    The first data set examined changes in cell phenotype before and 7 days after rhinovirus infection in healthy adults. The flow cytometric staining panel included markers of T cell differentiation and activation as well as rhinovirus-specific tetramers. The results of T-REX convincingly demonstrate "hotspots" that are expanded at 7 days and enriched for tetramer-staining cells. Thus, this study succeeds at demonstrating the utility of this method for identification of rare cells and the authors use this data set to appropriately determine the model parameters. Combining the results of this algorithm with the "Marker Enrichment Modeling (MEM)" method to characterize the markers expressed on those cell populations identified through T-REX is also very informative since this automates the characterization (that traditionally needs to be done by manual investigation).

    This first data set is relevant for this demonstration, but in some aspects it represents a best case scenario. "Phenotypic" identification of antigen-specific T cells in this way is only possible because the time point was chosen to capture the relatively narrow window when T cells would be activated, and there was access to a baseline sample for comparison. The authors do address the second point, and perform the analysis comparing day 7 to a later time point, day 28, as an appropriate alternative. The first concern limits the generalizability of this approach. In fact, the second example dataset examining mass cytometry data in patients with COVID-19 does in fact demonstrate limited ability to detect change in cell populations for many study participants.

  5. eLife

    Reviewer #1 (Public Review):

    Sierra M. Barone developed an automated, quantitative toolkit for immune monitoring that would span a wide range of possible immune changes, identify and phenotype statistically significant cell subsets, and provide an overall vector of change indicating both the direction and magnitude of shifts, either in the immune system as a whole or in a key cell subpopulation. The machine learning workflow Tracking Responders Expanding (T-REX) was a modular data analysis workflow including UMAP, KNN, and MEM. T-REX is designed to capture both very rare and very common cell types and place them into a common context of immune change. T-REX was analyzed data types including a new spectral flow cytometry dataset and three existing mass cytometry datasets.

    The conclusions of this paper are mostly well supported by data, but one aspect need to be clarified and extend. Cytometry tools like SPADE, FlowSOM, Phenograph, Citrus, and RAPID generally work best to characterize cell subsets representing >1% of the sample and are less capable of capturing extremely rare cells or subsets distinguished by only a fraction of measured features. Tools like t-SNE, opt-SNE, and UMAP embed cells or learn a manifold and represent these transformations as algorithmically-generated axes. The advantages of T-REX tool were not very clear.

  6. eLife

    Evaluation Summary:

    The three reviewers were overly positive about the stated goal of your T-REX method to characterize rare populations by cytometry. The potential applications in the context of analyzing antigen-specific T cells (as identified as tetramer-positive cells) were not missed on the reviewers and the use of the 2 timepoints-cohort of samples from rhinovirus-infected patients was judged clever. However, all three reviewers requested some edits and additional tests to really distinguish T-REX from other methods in terms of performance, and to better understand its analysis power. Reviewer #1 enjoined you to clarify the improvements of your method compared to previous methods. Reviewer #2 requested more stringent tests of your methods against functional datasets. Reviewer #3 inquired about corrections for batch effects, the result consistency for repeated down-sampling as well as the scalability of the method (especially when UMAP is being used).

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 and Reviewer #3 agreed to share their names with the authors.)

  7. ScreenIT

    SciScore for 10.1101/2020.07.31.190454: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Antibodies
    SentencesResources
    Cells were stained with antibodies that target markers of naïve, memory and helper T cells (CCR6, ICOS, CXCR3, CD27, CCR5, TBET, CD45RA, CD45R0, CD95, CXCR5, TCF1, CCR7), and activation and proliferation (CD25, CD38, CD127, Ki-67, PD-1).
    CCR6
    suggested: None
    ICOS
    suggested: (Leinco Technologies Cat# C2851, RRID:AB_2829607)
    CXCR3
    suggested: None
    CD27
    suggested: None
    CCR5
    suggested: None
    CD45RA
    suggested: None
    CD45R0
    suggested: None
    CD95
    suggested: (Leinco Technologies Cat# C1189, RRID:AB_2828354)
    CXCR5
    suggested: None
    TCF1
    suggested: None
    CCR7
    suggested: None
    CD25
    suggested: (BD Biosciences Cat# 560249, RRID:AB_1645496)
    CD38
    suggested: None
    CD127
    suggested: None
    Ki-67
    suggested: None
    PD-1
    suggested: None
    Software and Algorithms
    SentencesResources
    Data availability and transparent analysis scripts: Datasets analyzed in this manuscript are available online, including at FlowRepository 39.
    FlowRepository
    suggested: (FLOWRepository, RRID:SCR_013779)

    Results from OddPub: Thank you for sharing your code and data.

    Results from LimitationRecognizer: An explicit section about the limitations of the techniques employed in this study was not found. We encourage authors to address study limitations.

    Results from TrialIdentifier: We found the following clinical trial numbers in your paper:

    IdentifierStatusTitle
    NCT02796001Enrolling by invitationHallmarks of Protective Immunity in Sequential Rhinovirus In…

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  8. Version published to 10.1101/2020.07.31.190454v2 on bioRxiv
  9. Version published to 10.1101/2020.07.31.190454v1 on bioRxiv