SPICEiST: Subcellular RNA Pattern Enhances Cell Clustering of Imaging-Based Spatial Transcriptomics

Background: Imaging-based spatial transcriptomics (ST) enables the quantification of gene expression at single-cell resolution while preserving spatial context, but its utility is limited by small gene panels and challenges in accurate cell segmentation. To address these limitations, we present a graph autoencoder framework that integrates subcellular transcript distribution patterns with cell-level gene expression profiles for enhanced cell clustering in imaging-based ST (SPICEiST). Results: We systematically evaluated the clustering performance of SPICEiST across several cancer datasets and gene panel sizes. Our results demonstrate that the developed method consistently outperforms the conventional cell-level gene expression-based method in distinguishing subtle cell types, as measured by clustering indices, including CHI and DBI. Notably, SPICEiST reveals more spatially intermixed and less compartmentalized cell clusters, reflecting the complex and heterogeneous nature of tumor microenvironments. The improvement in cell clustering indices over the conventional approach was more pronounced in datasets with small gene panels of around 300 genes, in contrast to those with large panels containing over a thousand genes. Conclusions: These findings highlight the value of leveraging subcellular transcript patterns to overcome the inherent limitations of imaging-based ST, particularly for small gene panels, and may provide new insights into tumor heterogeneity.