Triage Advice From General-Purpose Language Models in Health and Everyday Situations

Read the full article See related articles

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

One adult in 10 has consulted a chatbot for health information within 6 months, and nearly half of users do so to decide whether they need care. Referral regardless of need could add demand on stretched services.

Objective

To compare referral to a professional by general-purpose language models in health and non-health situations against the level of contact that human raters judged necessary.

Design, Setting, and Participants

Cross-sectional study of 4,346 responses generated in September 2026 by 53 general-purpose model configurations, released between March 2023 and September 2026 and accessed through APIs or run locally, to 82 investigator-written messages (41 health and 41 non-health). No patients were involved.

Exposures

Health vs non-health scenario.

Main Outcomes and Measures

Professional referral, unconditional or conditional on a future change or on doubt, coded by 3 language models and compared with the level of contact that physicians, pharmacists and 2 technically trained raters judged necessary.

Results

Referral occurred in 74.7% of health and 46.3% of non-health responses, more often for health in every configuration (standardized difference, 28.3 percentage points; 95% CI, 18.3-37.0). Where raters judged contact unnecessary, 64.5% of health vs 27.6% of non-health responses recommended a professional (odds ratio, 17.08; 95% CI, 6.14-47.54); most of the health excess was conditional referral (46.3% vs 21.0%), and unconditional referral was 18.1% vs 6.6% (odds ratio, 4.66; 95% CI, 1.43-15.19). Where an in-person assessment was judged necessary, referral was similar (89.6% vs 84.5%). In medicines and physical health, referral was not associated with rated severity (odds ratio per point, 1.16; 95% CI, 0.53-2.55), unlike outside health (3.86; 95% CI, 1.79-8.33). The referral gap was 20.5 percentage points among configurations released in 2023 and 32.2 and 32.3 in 2025 and 2026 (4.8 percentage points per year; 95% CI, 2.3-7.2).

Conclusions and Relevance

General-purpose language models referred people to a professional where a visit was needed, in health as outside it, but in health also in 2 in 3 situations that physicians and pharmacists judged manageable without contact, a pattern stronger in newer models. Evaluations of these systems should include whether their advice distinguishes situations that can be deferred or managed at a distance, alongside the referral of serious cases.

Article activity feed