A Simulation-Based Validation Framework for Uncertainty-Aware Clinical Triage: Conformal Prediction and Ensemble Learning Applied to a Synthetic ICU Benchmark Cohort
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background and Objectives
Deploying machine learning for ICU triage requires simultaneously validated uncertainty quantification, demographic fairness, and interpretable attribution — three properties rarely assembled into a single pipeline. Controlled evaluation of such pipelines is difficult on real EHR data due to missing values, treatment confounding, and access restrictions. We develop and validate a complete uncertainty-aware triage pipeline in a simulation environment with known ground truth, providing pre-deployment evidence before application to real patient data.
Methods
We constructed a SOFA-calibrated synthetic ICU cohort ( N = 90,000; 29.2% mortality) with marginal distributions grounded in published MIMIC-III and eICU epidemiology [4, 29]. Fourteen classifiers were trained, including XGBoost, LightGBM, and a custom FT-Transformer tuned by Bayesian optimisation [26]. Seven composite features were engineered from clinical first principles, including a novel lactate/albumin ratio ( r LA ). The uncertainty quantification pipeline comprised Monte Carlo Dropout decomposition into epistemic and aleatoric components [9], distribution-free conformal prediction at three miscoverage levels [11, 12], and selective prediction with principled abstention [13]. Framework outputs were routed through a three-zone clinical triage system, with ablation, feature importance, temporal consistency, and demographic fairness evaluation.
Results
All 14 classifiers exceeded AUC 0.85 on the balanced test split ( n = 25,073). XGBoost achieved AUC 0.967 (95% CI 0.965–0.970) versus SOFA AUC 0.731 on the natural-distribution held-out cohort ( n = 18,000; 29.3% mortality); this gap quantifies signal-recovery efficiency within the simulation, not clinical superiority. Conformal coverage matched the theoretical guarantee at all tested miscoverage levels. Selective prediction raised AUC from 0.917 to 0.980 at 50% abstention. The UNCERTAIN triage zone reached AUC 0.624 (chance level), confirming the uncertainty quantile correctly isolates ambiguous cases. Temporal AUC variation was 0.003; sex and age fairness gaps were 0.005 each.
Conclusions
The integrated pipeline behaves as theoretically expected under known ground-truth conditions, validating each component in isolation and in combination. The simulation provides the methodological foundation for a planned MIMIC-IV validation study. Code and cohort generation scripts will be deposited on Zenodo upon acceptance.
