Rule-Based Electronic Sepsis Alerts Identify High-Risk Patients Despite Poor Diagnostic Accuracy: A Real-World Evaluation and Implications for Machine Learning

Objective To evaluate the diagnostic accuracy of an electronic sepsis alert system (ESAS) in an acute care hospital within an electronic medical record (eMR) system. Design Single-centre observational study of prospectively collected data from the eMR incorporating a third-party electronic sepsis surveillance and alerting system. Clinical eMR and administrative coding data for all patient records were analysed. Performance characteristics of the ESAS were compared with the presence or absence of clinical sepsis. Setting A university-affiliated hospital in Melbourne, Australia with 25,000 multiday-stay admissions per annum. Participants All adult multiday-stay admissions between January 1 ^st , 2018, and December 31 ^st , 2019, inclusive. Main Outcome measures Sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) of the ESAS. Results 149,053 records were included in the study, of which 4,011 triggered an electronic sepsis alert. The sensitivity and PPV of the ESAS were 26.3% [95% CI, 25.1-27.6%] and 33.2% [95% CI, 31.7-34.7%] respectively, while its specificity and NPV were 98% [95% CI, 98.0-98.1%] and 97.3% [95% CI, 97.2–97.4%] respectively. Conclusion The ESAS was highly specific but lacked sensitivity for reliable clinical application. The activation of ESAS was associated with a longer length of stay, higher rates of Intensive Care Unit admission and in-hospital mortality. The ESAS ultimately identified a cohort at risk of clinical deterioration. These results highlight fundamental limitations of rule-based approaches and underscore the need for adaptive machine learning systems that can better integrate complex clinical patterns for early sepsis detection.