Multi-view deep learning of highly multiplexed imaging data improves association of cell states with clinical outcomes

Highly multiplexed imaging technologies can quantify the expression of 10s-100s of proteins and their modifications at sub-cellular resolution. Applications to tissue specimens across a broad array of pathologies have enabled patient subtyping in terms of novel cell states and spatial cell communities and linked their prevalence and overall architecture to patient outcomes. While highly multiplexed imaging analysis workflows typically summarize each cell in terms of its post-segmentation mean expression, additional cellular information can be quantified including cell morphology, sub-cellular expression patterns, and spatial cellular context, ultimately giving a multi-modal view of each cell. While deep latent variable models such as variational autoencoders (VAEs) are well established for other multi-modal single-cell assays, their ability to integrate these multiple views of a cell from highly multiplexed imaging data remains largely unknown.

Here, we explore the abilities of multi-modal VAEs to learn unified latent cellular representations from multiple views of each single-cell quantified from highly multiplexed imaging, including mean expression, morphology, sub-cellular protein co-localization, and spatial cellular context, while conditioning on technical and batch specific effects. We first discuss strategies for training and hyperparameter optimization of such models across a set of highly multiplexed imaging datasets of breast and melanoma cancers. We next quantify the relevance of the integrated multi-modal latent space in predicting patient-specific clinical outcomes and demonstrate competitive performance across a range of outcomes compared to existing baselines. Then, we explore the ability of the cellular representations to learn cellular phenotypes that align with known cell types and find that the expression-specific latent representation can identify clusters more closely aligned with known lymphocyte and epithelial subpopulations compared to established workflows. Finally, we test the ability of each representation to perform consistent batch integration across different cohorts and discuss the considerations on taking known background variables into account.