A Methodology to Evaluate the Efficacy and Patient-Centeredness of a Patient-Facing AI Chatbot: Development and Validation of a Standardized-Patient Method
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Patient-facing AI intake chatbots can enter clinical use without regulatory review, so their errors may first surface with actual patients.
Objective
To develop and pilot a remote, voice-based standardized patient (SP) method for evaluating an intake chatbot’s history-taking, documentation, and patient-centeredness.
Methods
Four primary care cases had Easy and Difficult variants. In Phase 1, five SPs completed 40 encounters with internal medicine residents rating portrayal fidelity. In Phase 2, SPs completed 80 encounters with a ChatGPT 4o chatbot, recorded whether critical elements were asked about and addressed satisfactorily, and rated summaries; four debriefing interviews were analyzed. In Phase 3, two blinded clinicians rated transcripts and summaries from 101 encounters (32 resident-led, 69 chatbot-led).
Results
Portrayal fidelity averaged 4.52 of 5. The chatbot asked about 59.3% of critical elements in Easy versus 50.2% in Difficult variants (adjusted OR 1.49; 95% CI, 1.15 to 1.93) and addressed 73.3% versus 64.6% (OR 1.47; 1.12 to 1.93); near-ceiling summary quality did not differ. SPs reported missed information and rushed pacing. Reviewer agreement was moderate to substantial for element judgments (AC1 0.49 to 0.70) but poor for totals (ICC 0.10 and 0.21). In exploratory comparisons, residents asked about more elements (adjusted difference 12.5 percentage points; 95% CI, 7.0 to 18.1) and satisfactorily addressed more (9.2 points; 3.8 to 14.6); totals did not differ.
Conclusions
SP portrayal was consistent enough for pre-clinical chatbot evaluation, and the method detected lower coverage in harder cases. Documentation ratings alone may miss conversational gaps, which require direct measurement.
