Language-dependent diagnostic safety of medical AI systems: a cross-lingual benchmarking and prospective clinical study

Read the full article

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

Background

Patients worldwide receive healthcare in many languages, yet medical AI systems are validated almost exclusively in high-resource languages such as English and Chinese, exposing patients in other linguistic settings to unquantified diagnostic risk. Existing multilingual evaluations rely on translated research-style benchmarks that fail to capture authentic clinical work. We aimed to characterise the patient safety consequences of multilingual medical AI deployment in real-world clinical settings and to develop an auditable detection method for unsafe outputs.

Methods

We evaluated different language models(LLMs) and visual language models(VLMs) across four real-world clinical tasks (conversational QA, radiology report generation, glaucoma diagnosis, ICU re-intubation prediction) in five languages (English, Chinese, Malay, Thai, Persian). We developed a token-level uncertainty toolkit to localise reasoning instability, com pared three inference paradigms (native-language, English chain-of-thought, back-translation pivot), and conducted a prospective study (50 dialogues, 150 physician-reviewed records).

Findings

LLMs/VLMs performance degraded consistently from high-to low-resource languages across al l tasks. Key gaps included: HealthBench score declining from 0·3743 to 0·3180; radiology macro-F1 from 0·2938 to 0·2149–0·2424, consistent with selective pathology suppression; glaucoma accuracy from 50·7% to 32·7%; ICU parameter recall from 100·0% to 48·5%. Multimodal inputs amplified degradation. Qwen3 VL 235B showed attenuated decline with no re source-ordered pattern in glaucoma classification. Token-level analysis localised instability to mid-chain stages (40–70% of the normalised trajectory); perplexity-based confidence failed to flag errors (AUROC 0·41–0·66). Back-translation pivot consistently restored performance. In the prospective study, 98·7% of records required physician edits (overall modification score 53·6%); Thai-pivot correction burden (59·0%) exceeded English-pivot (5 0·7%, p=0·003) and Chinese-direct (51·0%, p=0·004).

Interpretation

Multilingual deployment produced clinically consequential failures, including missed pathology, distorted physiological extraction, and amplified multimodal misclassification, that were invisible to monolingual validation and not reliably flagged by model confidence. Pre training data composition may contribute to multilingual safety risk. Language-specific safety auditing should precede deployment in non-dominant-language healthcare settings; the open-source detection toolkit enables this without model retraining.

Article activity feed

  1. medRxivが、医療AIの診断安全性に「言語差」が出る可能性を報告。5言語・4つの実臨床タスクで、翻訳ベンチマークだけではなく多言語環境での直接的な安全性評価が必要だと示しています。 https://www.medrxiv.org/content/10.64898/2026.05.19.26353490v1

  2. medRxiv reports language-dependent diagnostic safety risks in medical AI. Four real-world tasks across five languages showed that multilingual deployment needs direct safety testing, not just translation-based benchmarks. https://www.medrxiv.org/content/10.64898/2026.05.19.26353490v1

  3. Language-dependent diagnostic safety of medical AI systems: a cross-lingual benchmarking and prospective clinical study https://www.medrxiv.org/content/10.64898/2026.05.19.26353490v1