Sparse Foundation Models for Continous-Time EHRs

Electronic health records (EHRs) are characterized by extreme sparsity, irregular sampling, and event-driven observation processes, yet most existing foundation models treat them as dense sequences through discretization and tokenization. This mismatch forces models to infer temporal dynamics implicitly through depth and attention, conflating missingness with semantic content and limiting robustness under real-world sparsity. We propose a Sparse Foundation Model (SFM) that represents patient trajectories as continuous-time latent states governed by neural ordinary differential equations, with event-conditioned residual updates applied only at observed times. This formulation unifies continuous-time dynamics with ResNet-style depth, yielding a sparsity-adaptive architecture whose effective depth is determined by the data rather than a fixed temporal grid. We pretrain SFM using self-supervised objectives defined over irregular event streams, enabling the model to reason explicitly about missing observations and variable temporal gaps. Across MEDS-DEV and FoMoH benchmarks, SFM consistently outperforms state-of-the-art EHR foundation models, including CEHR-BERT, CEHR-GPT, CLMBR, and MOTOR, with particularly large gains under extreme sparsity, limited labeled supervision, and unseen prediction horizons. Beyond accuracy, SFM exhibits improved calibration, temporal stability, and robustness to observation dropout. These results suggest that aligning foundation model architecture with the sparse, continuous-time structure of clinical data is critical for scalable and reliable healthcare representation learning.