spEMO: Exploring the Capacity of Foundation Models for Analyzing Spatial Multi-Omic Data

Several pathology foundation models have been designed by pre-training a model with pathology information for disease-centered downstream applications. These models have been treated as a breakthrough for pathology research. Along with pathology images, we can also measure spatial multi-omic expression levels for each spot, which provide additional information for understanding the spatial context. However, we still lack an effective tool to leverage the contributions of these technologies. Here, we present a novel artificial intelligence system, named as spEMO, to incorporate the embeddings from pathology foundation models and large language models to analyze spatial multi-omic data. Overall, spEMO outperforms foundation models trained only with single-modality data through introducing better representations. Our method also explores different approaches to combining information from various sources and shows the contributions of integrating external embedding information to handle different novel downstream applications, including spatial domain identification, spot-type prediction, whole-slide disease-state prediction and interpretation, multi-cellular interaction inference, and medical report generation. Furthermore, we define a new task of multi-modal alignment to assess the information retrieval ability of pathology foundation models. This task offers a new direction to evaluate the quality of foundation models and gain insights for model development.