A Physically-Realistic Simulator and Cosine-Based Decoder for MERFISH Spatial Transcriptomics

Imaging based spatial transcriptomics technologies such as MERFISH have opened new avenues for studying cellular organization and gene expression within intact tissues. However, the accuracy of downstream analyses depends critically on the decoding step that reconstructs barcodes from fluorescence patterns and maps them to gene identities. Despite a growing number of decoding methods, limited systematic benchmarking has made it difficult to assess accuracy, sensitivity, and robustness under varying experimental conditions. Here, we introduce Serval, which is a modular framework for evaluating and comparing decoding methods in spatial transcriptomics. Serval separates key decoding stages into independently configurable modules, enabling flexible integration of alternative algorithms. Using the Serval framework we develop a novel decoding method that improves transcript recovery by optimizing the cosine similarity to known barcodes. To evaluate performance under controlled conditions, we developed a photorealistic multiplexed spot simulator that generates MERFISH-like images with ground-truth labels. Using both simulated and real datasets, including cultured cells and tumor sections, we show that the new cosine decoder achieves higher correlation with bulk and single-cell RNA-seq references compared to existing methods. Our results demonstrate that modular decoding frameworks combined with reproducible benchmarking tools can guide method development and support more accurate spatial transcriptomics analysis in complex biological samples.