scRM: A single-cell foundation model trained on scRNA-seq reads
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Single-cell foundation models currently learn from gene count matrices, which are discretized, species-specific summaries that discard the RNA sequence, splice-site usage, 3 ′ UTR isoform identity and the mutational information present in the underlying sequencing reads. Here we introduce single-cell raw data pretraining, a framework that trains foundation models directly on sequencing reads, requiring no read alignment or count matrix, allowing any trained model to be applicable to arbitrary species.
Our first model in this framework, the single-cell read model scRM, pairs two transformer encoders with a hierarchy of attention pools and contains 1.2 million parameters. We trained it as a JEPA on 3,000 mouse neuron cells on a single H100 in one hour, and benchmarked it zero-shot against leading single-cell foundation models with orders of magnitude more parameters and training cells. Our results show that our lightweight model reaches the performance of existing models on all tasks, and exceeds every one of them on perturbation detection tasks whose detection requires sequence-level detail.
To our knowledge, scRM is the first single-cell foundation model trained on sequencing reads, demonstrating that raw data pretraining is not only feasible but competitive at a fraction of the computational cost. We anticipate that scaling this framework and exploring alternative architectures will unlock biological information inaccessible to count models and extend the dataset-specific pretraining it already makes possible. The source code is available at https://github.com/altoslabs/scRM .