A deep learning-based multiscale integration of spatial omics with tumor morphology

Spatial Transcriptomics (spTx) offers unprecedented insights into the spatial arrangement of the tumor microenvironment, tumor initiation/progression and identification of new therapeutic target candidates. However, spTx remains complex and unlikely to be routinely used in the near future. Hematoxylin and eosin (H&E) stained histological slides, on the other hand, are routinely generated for a large fraction of cancer patients. Here, we present a novel deep learning-based approach for multiscale integration of spTx with tumor morphology (MISO). We trained MISO to predict spTx from H&E on a new unpublished dataset of 72 10X Genomics Visium samples, and derived a novel estimate of the upper bound on the achievable performance. We demonstrate that MISO enables near single-cell-resolution, spatially-resolved gene expression prediction from H&E. In addition, MISO provides an effective patient representation framework that enables downstream predictive tasks such as molecular phenotyping or MSI prediction.