Clinical referral text overrides MRI evidence in multimodal large language models: glioma versus brain abscess
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objectives
To determine how strongly clinical referral text, rather than image content, drives the decisions of multimodal large language models (LLMs) differentiating glioma from brain abscess on MRI.
Materials and methods
This retrospective single-center study included 60 contrast-enhanced T1-weighted brain MRI examinations (30 gliomas, 30 abscesses; reference-standard diagnosis). One axial slice per case was paired with five referral texts (explicitly inflammatory to explicitly tumor-suggestive) under three prompt instructions and submitted 20 times each to GPT-4o, Claude Sonnet 5, Gemini 3.5 Flash, and Qwen 3.6 (72,000 answers). The primary outcome was the text-driven span (difference in tumor-call frequency between explicit inflammatory and explicit tumor text). Two blinded radiologists classified the images without text; sensitivity, specificity, and accuracy were compared descriptively.
Results
Patients (29 women) had a mean age of 55.8 ± 16.4 years. With neutral text, models answered tumor in 45–67% of queries (specificity, 0.38–0.78), versus 52% and 60% for radiologists (specificity, 0.77 and 0.63). The text-driven span was 0.79–1.00 ( p < 0.001 for all models), whereas confirmed gliomas and abscesses differed by only 0.01–0.10 in three models and 0.25 in Gemini 3.5 Flash. Subtle wording produced at least 83% of the span in three models. An instruction to ignore the text reduced the span to 0.48–0.70 in three models and to 0.04 in Gemini 3.5 Flash, partly through reversion to a tumor default. Model accuracy was 0.52–0.73 versus 0.78 and 0.73 for radiologists.
Conclusion
Multimodal LLMs deferred to referral text over imaging evidence in differentiating glioma from brain abscess, and prompt instructions did not reliably correct this.
Key Points
Question
Multimodal large language models receive images together with clinical referral text, but how strongly the text rather than the image determines their diagnosis is unknown.
Findings
Changing only the referral text largely reversed the glioma-versus-abscess decision for identical MRI images in four models; instructions to ignore the text reduced this inconsistently.
Clinical relevance
Before clinical use, multimodal large language models require model- and task-specific testing of their susceptibility to referral text, because instructing them to prioritize the image is not a dependable safeguard against text-driven misdiagnosis.
