Evaluating agentic simulation for local public health modelling

Large language model (LLM)-based generative agents can reproduce aspects of individual human behavior, but whether they can be scaled to geographically grounded populations that reproduce real-world health behaviors remains unclear. Here, we introduce LLMPopSim, an agentic population simulation framework that uses U.S. Census and Centers for Disease Control and Prevention data to construct populations of AI agents whose simulated individual health behaviors can be aggregated and evaluated at the community level. We developed the framework using historical data from Hawaiʻi in 2018, and evaluated temporal and geographic generalizability using held-out 2022 cohorts from Hawaiʻi and New York State, with colorectal cancer screening and mammography as proof-of-concept behaviors. Across the four state-outcome evaluations, mean absolute error ranged from 3.5 to 15.0 percentage points and correlations between simulated and observed ZCTA-level prevalence ranged from 0.26 to 0.69. Mean prediction bias was +15.0 and +10.2 percentage points for colorectal cancer screening in Hawaiʻi and New York State, respectively, and +2.9 and +0.9 percentage points for mammography; ratios of predicted to observed geographic standard deviation were 0.60 and 0.68 for colorectal cancer screening and 1.12 and 1.96 for mammography, respectively. Prediction error was greatest in communities with lower observed screening prevalence and varied across community characteristics without a uniform socioeconomic gradient. These findings demonstrate that individually represented AI agents can aggregate into population-level patterns that retain measurable features of real-world health behavior across temporal and geographic transfer, while identifying calibration, distributional fidelity and subgroup performance as key challenges for generative population simulation. LLMPopSim provides an empirical foundation for developing synthetic populations that may ultimately enable simulation of heterogeneous population responses to public health interventions.