Benchmark Self-Triage (BeST): Comparing Self-Triage Systems and Multiple Generations of OpenAI Models Against Preliminary Expert Consensus
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
As digital self-triage becomes more widely used, quality assurance requires evaluation of both emergency recognition and overall agreement with clinical judgement, including how these outcomes differ across generations of large language models. The Benchmark Self-Triage (BeST) project intends to provide a framework to compare systems for digital self-triage against a medical 'gold standard' in Germany. As a first step in developing and demonstrating the methodology, we compared study-specific implementations of SmED 5.4.11 accessed via API and Infermedica Engine API via Microsoft Azure Healthcare Agent Service with five structured OpenAI configurations spanning multiple model generations: gpt-4.1, gpt-5.4, gpt-5.6-luna, gpt-5.6-sol, and gpt-6-astra. The manufacturers of SmED and Infermedica offer MDR-certified Class IIb triage products, but these certified products were not directly evaluated. An LLM-based patient actor covered 480 physician-curated vignettes derived from German routine triage data. A pool of seven physicians provided preliminary reference classifications and consensus for 480 vignettes. Recommendations were mapped to three urgency categories-emergency, acute, and not urgent-and evaluated using winsorised weights derived from routine triage frequencies. Weighted emergency sensitivity ranged from 51.3% (95% CI 35.5-66.3%) for gpt-4.1 to 99.9% (99.7-100.0%) for gpt-6-astra. Weighted exact three-category agreement (accuracy) ranged from 54.4% to 68.8%; gpt-4.1 achieved the highest agreement despite the lowest emergency sensitivity. No system performed best across all urgency categories. The later-generation OpenAI configurations showed a more precautionary triage profile resembling SmED, with higher sensitivity for emergency cases; gpt-6-astra showed the strongest shift. This cross-sectional comparison across model generations suggests a shift towards more precautionary triage behaviour, without establishing a longitudinal improvement in accuracy. These findings highlight the limitations of overall agreement for evaluating self-triage and support category-specific assessment. Broader expert consensus and evaluation, as well as further steps to reduce dependence on SmED-derived source data, are needed and planned before drawing conclusions about clinical safety.