Can Language Models Diagnose Sepsis? A Head-to-Head Evaluation Against 353 Physicians
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Importance
Sepsis is a leading cause of in-hospital mortality, where timely and accurate diagnosis is critical yet frequently delayed. Whether large language models (LLMs) can diagnose sepsis reliably — and whether they do so fairly, consistently, and with well-calibrated confidence — remains unknown.
Objective
To evaluate the diagnostic capabilities of LLMs on standardised sepsis vignettes before and after biomarker disclosure, benchmarked against physicians, and to characterise their robustness, stability, bias and confidence calibration.
Design
Retrospective in-silico study based on six anonymised real cases spanning the diagnostic spectrum from no infection to septic shock. For each case, the reference-standard diagnosis was assigned, following the Sepsis-3 framework. Each case was evaluated under a two-stage (pre– and post–biomarker disclosure) protocol. To assess potential demographic bias, every case was additionally presented while systematically varying the patient’s sex, age, and language.
Setting and Participants
Five locally deployed LLMs (DeepSeek-4, Qwen3.8-27B, its quantised variant, Gemma-4-31B, and the domain-specific MedGemma) were evaluated in a controlled evaluation pipeline and compared with responses from 353 clinicians, using identical clinical vignettes.
Main Outcomes and Measures
Diagnostic accuracy before/after biomarker disclosure, per-vignette error, sensitivity to demographic attributes (diagnosis-change rate) and its directional component, reproducibility across repetitions and confidence calibration.
Results
Before biomarkers, model accuracy ranged 40–67%, comparable to physicians (57%). After disclosure, physician accuracy was 61%, exceeded by Qwen3.8-27B (91%), its quantised variant (86%) and Gemma-4-31B (83%); DeepSeek-4 reached 64% and MedGemma 56%, the only model below physicians. Biomarkers rescued confirmatory cases (pancreatitis-associated sepsis), whereas post-surgical sepsis remained difficult for models and physicians alike. Quantisation cost only 5 accuracy points. Patient sex and age showed no directional bias, and language was the only bias. Gemma-4-31B was the most reproducible model, the Qwen models were the most accurate and best calibrated, and MedGemma the least.
Conclusions and Relevance
LLMs, including quantised variants, can provide accurate informed decision support for sepsis, surpassing physicians once laboratory data are available. However, robustness, calibration, and language fairness varied independently of accuracy and should guide model selection and clinical validation.
