Multi-agent LLM teams improve differential recall on curated cases but underperform on emergency working-label matching: a two-cohort, two-family evaluation
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Whether multi-agent large language model (LLM) pipelines help, add nothing, or underperform a single call may depend on the input and scoring contract rather than the architecture; these boundary conditions are unmapped.
Methods
We compared a single direct call (A×1), five personas in one context (P), and five isolated roles integrated by a moderator (MDT) across curated corpora (87 clinicopathological cases, 406 MedCaseReasoning cases, 392 replication vignettes) and two disjoint emergency cohorts from one repository (364 mixed-label encounters; a protocol-frozen 400-encounter disease-label-only replacement, one local serving stack). Scoring was judge-mediated top-k working-label matching. A completed factorial isolated the active ingredient; a symptom-permissive output-contract arm tested contract fit; three physicians rated discordant cases.
Findings
On curated corpora MDT improved differential-list recall (top-3: +5·5 points, p = 0·0079; +3·0, p = 0·0079; +3·3, p = 0·0013) but not top-1 accuracy; the factorial attributed the gain to independent generation plus moderated synthesis, not personas. On emergency presentations the sign reversed: in the disease-label-only cohort A×1 led MDT at every endpoint (top-3 −5·2 points, 95% CI −8·0 to −2·4, p = 0·0008), a second family reproduced it, and bridged cases excluded the serving stack. A symptom-permissive contract shrank the deficit’s point estimates by 52–75%, replacement-cohort intervals all spanning zero though remaining negative; the clause harmed no arm. Physician safety ratings were inconclusive (original-cohort gate missed; replacement-cohort rescue contrast unadjudicable, 95% CI −32·7 to +2·6).
Interpretation
Multi-agent diagnostic gains are real but conditional: the sign of the effect tracks input genre, and its magnitude tracks output-contract fit. Curated-benchmark gains should not be extrapolated to first-contact emergency text, where role-based teams underperform a single call at seven-fold token cost. The reversal is label-level and judge-mediated; clinical safety is neither established nor excluded.
Funding
None.
Research in context
Evidence before this study
Multi-agent LLM diagnostic pipelines report gains, but specialist personas, independent generation, synthesis, and additional test-time compute are usually varied together, so the active ingredient is unidentified. Evaluation is concentrated on curated case reports, leaving performance on first-contact notes and working-diagnosis labels unclear, and no study has tested whether the output contract demanded of the model contributes to measured underperformance.
Added value of this study
Across curated corpora and two disjoint emergency cohorts, we show the multi-agent effect is conditional and map two of its moderators. A completed factorial attributes the curated gain to independent generation plus moderated synthesis, not personas, and confines it to differential-list recall. The emergency deficit is established on a protocol-frozen cohort whose labels are all disease-level diagnoses, reproduced by a second model family on the same cases, and insensitive to the serving stack on bridged cases—so neither symptom-level labels nor the stack explains it. A symptom-permissive output contract closes the breadth component of the deficit while the ordering component persists. Physician safety ratings are inconclusive by design.
Implications of all the available evidence
Gains measured on curated benchmarks should not be extrapolated to first-contact clinical text. Where multi-call designs are used on uncurated input, plain independent samples with a moderating call avoid the role-list penalty. An output contract permitting undifferentiated symptom-level entries is free—it harmed no arm—and is a defensible default wherever reference labels may be working diagnoses. Strategy selection by input, model, and endpoint remains a hypothesis requiring prospective evaluation, with independent emergency datasets and adequately powered safety studies.
