Generative embedding of sparse data with a tabular foundation model for dengue anticipatory action: a machine learning approach
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Early outbreak detection has largely relied on complex, data-intensive models with limited applicability to low-resource surveillance. Even state-of-the-art tabular foundation models require dense datasets for fine-tuning to capture disease transmission dynamics. We address this by building a domain-mechanistic generative embedding from cases and rainfall to detect early epidemic onset.
Methods
We build a generative, domain-mechanistic embedding from sparse case and rainfall data into 132 features, converting limited inputs into a structured representation of transmission for outbreak-onset detection. A tabular foundation model was evaluated by leave-one-year-out validation with cluster-bootstrap intervals across 17 Philippine regions and eight dengue-endemic countries, benchmarked against raw data columns and catch22.
Findings
Raw columns used as input to the tabular foundation model were weakly predictive of dengue outbreak onset (AUROC 0·56–0·70). The generative embedding improved detection to 0·77 across countries and 0·89 across regions (+0·205 and +0·183; paired cluster-bootstrap p≤0·006). Calibration error was lower at the regional scale than at the country scale (expected calibration error 0·067 and 0·149). Strongly seasonal regions and countries were the most predictable (Philippine Type I region mean 0·87; Mexico 0·94, Brazil 0·93, the Philippines 0·91), whereas countries with year-round or coastally opposing rainfall were weaker or below chance (Singapore 0·69, Sri Lanka 0·42), and countries left with only one or two seasons after applying the onset rule gave unreliable estimates.
Interpretation
Under sparse surveillance conditions, predictive capacity depended strongly on the representation supplied to the tabular foundation model. The generative embedding translates climate and epidemiological variables into actionable early-warning signals by capturing underlying transmission mechanisms, whose accuracy scales with local seasonal dynamics. This approach provides a viable pathway for extending prospective outbreak surveillance in data-limited settings, and indicates that mechanism-grounded embeddings could calibrate transmission-acceleration models at aggregated scales to improve their predictions.
Funding
National Institute of Environmental Health Sciences, National Institutes of Health (award P20ES036118).
Research in context
Evidence before this study
We searched PubMed, Web of Science, and Google Scholar up to April 2026, combining “dengue” with “early warning”, “outbreak prediction”, or “forecasting”, and “machine learning”, “foundation model”, or “feature engineering”. Foundation models have recently been applied to epidemic modeling across pathogens, including dengue, forecasting incidence from raw surveillance series with minimal retraining. Their main limitation is the absence of a disease transmission mechanism in an otherwise black-box model. Closing that gap has been framed as a problem of scale, requiring large-scale epidemic-specific datasets for retraining.
Added value of this study
We show that for sparse dengue surveillance, predictive capacity is set by how the input represents transmission dynamics, and that the transmission mechanism absent from a foundation model can be supplied through a generative embedding. Computing 132 features from these quantities lifted the tabular foundation model from near-chance and weak detection on the raw columns at the aggregated country and regional scales (AUROC 0·56 and 0·70) to stronger detection of the epidemic onset phase, with the clearest operational performance at the regional scale (AUROC 0·89). Regional scores detected epidemic onset, but calibration slopes below one showed that absolute probabilities require local recalibration before use. As climate change outpaces the build-out of surveillance, the capacity to anticipate outbreaks from counts and weather alone may matter most where data are limited.
Implications of all the available evidence
Recent benchmarks proposed adding disease mechanisms to foundation models through fine-tuning. A generative embedding of the transmission mechanism achieved strong detection performance on sparse data with no model retraining. The regional scale is the appropriate calibration and deployment scale for dengue anticipatory action. Moreover, the country scale is informative as a transportability test because heterogeneous reporting systems, asynchronous climate zones, and sparse retained seasons can weaken or invalidate national estimates. This principle, reconstructing a mechanism-grounded representation so a general-purpose model can process transmission, could extend to other climate-sensitive diseases that record little more than counts and weather.
