Diagnosis, triage and response quality of five large language models on emergency medicine vignettes: a blinded comparison
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Emergency care requires appropriate triage and management as well as diagnosis. We compared five large language models on these tasks using the same clinical vignettes.
Methods
Seventeen synthetic emergency department vignettes were submitted to Claude Fable 5, GPT-5.6 Sol, Grok 4.5, Gemini 3.6 Flash and Kimi K3 through consumer interfaces on 11–12 August 2026. Two emergency physicians independently assessed the 85 responses with model identities concealed. Outcomes were agreement with reference diagnoses and Canadian Triage and Acuity Scale (CTAS) levels, appropriateness and overall quality (1–5 scales), potential harm and preferred response.
Results
Every model correctly named the reference diagnosis in all 17 cases. CTAS exact agreement ranged from 14/17 to 15/17 per model (p=0.938). Of 13 discordant assignments, nine were undertriage and four overtriage. Appropriateness and quality differed across models (both p<0.001): Kimi K3 had the highest mean scores (4.82 and 4.74) and Grok 4.5 the lowest (3.91 and 3.71); Grok 4.5 scored significantly lower than every other model on both scales after Holm adjustment. Nine responses were flagged for potential harm: four from Grok 4.5, two each from Gemini 3.6 Flash and GPT-5.6 Sol, one from Claude Fable 5 and none from Kimi K3 (p=0.119).
Conclusion
In this selected set of synthetic vignettes, all five models correctly named the reference diagnosis, yet undertriage and potentially harmful omissions occurred and response quality differed between models. These findings do not establish any model’s clinical safety.
Key messages
What is already known on this topic
Large language models have been evaluated on emergency diagnosis, triage and management, including multi-model comparisons using the Canadian Triage and Acuity Scale.
What this study adds
In 17 synthetic vignettes, five models all named the reference diagnosis, yet physician ratings differed and triage errors and potential-harm flags remained.
How this study might affect research, practice or policy
LLMs, once clinically validated, have great promise to improve emergency care. Evaluations of these tools should assess management and urgency, not diagnosis alone. These findings do not establish that any model is safe for clinical use.
