Triage Advice From General-Purpose Language Models in Health and Everyday Situations
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
One adult in 10 has consulted a chatbot for health information within 6 months, and nearly half of users do so to decide whether they need care. Referral regardless of need could add demand on stretched services.
Objective
To compare referral to a professional by general-purpose language models in health and non-health situations against the level of contact that human raters judged necessary.
Design, Setting, and Participants
Cross-sectional study of 4,346 responses generated in September 2026 by 53 general-purpose model configurations, released between March 2023 and September 2026 and accessed through APIs or run locally, to 82 investigator-written messages (41 health and 41 non-health). No patients were involved.
Exposures
Health vs non-health scenario.
Main Outcomes and Measures
Professional referral, unconditional or conditional on a future change or on doubt, coded by 3 language models and compared with the level of contact that physicians, pharmacists and 2 technically trained raters judged necessary.
Results
Referral occurred in 74.7% of health and 46.3% of non-health responses, more often for health in every configuration (standardized difference, 28.3 percentage points; 95% CI, 18.3-37.0). Where raters judged contact unnecessary, 64.5% of health vs 27.6% of non-health responses recommended a professional (odds ratio, 17.08; 95% CI, 6.14-47.54); most of the health excess was conditional referral (46.3% vs 21.0%), and unconditional referral was 18.1% vs 6.6% (odds ratio, 4.66; 95% CI, 1.43-15.19). Where an in-person assessment was judged necessary, referral was similar (89.6% vs 84.5%). In medicines and physical health, referral was not associated with rated severity (odds ratio per point, 1.16; 95% CI, 0.53-2.55), unlike outside health (3.86; 95% CI, 1.79-8.33). The referral gap was 20.5 percentage points among configurations released in 2023 and 32.2 and 32.3 in 2025 and 2026 (4.8 percentage points per year; 95% CI, 2.3-7.2).
Conclusions and Relevance
General-purpose language models referred people to a professional where a visit was needed, in health as outside it, but in health also in 2 in 3 situations that physicians and pharmacists judged manageable without contact, a pattern stronger in newer models. Evaluations of these systems should include whether their advice distinguishes situations that can be deferred or managed at a distance, alongside the referral of serious cases.