Ambient AI Documentation in Mixed-Language Encounters: A Heuristic Evaluation of Reenacted Mandarin–English and Spanish–English Clinical Conversations

Ambient AI documentation systems rely on automatic speech recognition to transcribe patient–provider conversations before generating clinical notes. However, little evidence exists on how these systems perform in mixed-language clinical encounters. We conducted a mixed-methods heuristic evaluation of an ambient AI documentation tool using 24 reenacted primary care conversations, including 12 Mandarin–English conversations developed from real-world encounter excerpts and 12 Spanish–English adapted counterparts. Quantitative analyses measured mixed error rate (MER) and code-switching. Overall MER was low, with a median of 4% and less variation in Spanish–English conversations, and 9% in Mandarin–English conversations, but with outliers reaching 67%. The system generally detected language switches reliably, although deletions occurred frequently in Mandarin–English transcripts at switch points. Qualitative analysis revealed transcription errors related to phonetic similarity, automatic translation, clinical terminology recognition, and language-specific challenges. These findings highlight considerations for improving ambient AI tools to support multilingual providers in delivering care for linguistically diverse populations.