Quality and transcript-grounded clinical-risk error burden of AI-generated vs allied health professional clinical notes: a paired simulation study across four disciplines
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Ambient artificial-intelligence (AI) scribes are being deployed beyond medicine into allied-health practice, but comparative evidence mainly concerns physician documentation.
Objective
To compare quality and transcript-grounded clinical-risk error burden of AI-generated and allied health professional notes from identical simulated consultations.
Methods
Paired simulation across dietetics, physiotherapy, occupational therapy and speech and language therapy (five consultations each; 20 pairs; 40 notes). Heidi generated an unedited first-draft note and a professional documented each encounter. Professionals rated PDQI-9 quality. Professional and automated review identified unsupported statements and omissions; two language models graded clinical risk. Co-primary outcomes were PDQI-9 score and Critical+High error density.
Results
Mean PDQI-9 scores were 38.45 and 35.05 for AI and professional notes (paired difference +3.40, 95% CI +0.01 to +6.79; Wilcoxon p=0.042; dz=0.47). The Bayesian estimate was +3.47 (94% equal-tailed credible interval +0.66 to +6.30; P[AI>professional]=0.989). Automated-reviewer all-error density was 3.35 vs 8.55 per note (professional-minus-AI difference 5.20, 95% CI 3.85 to 6.55); omission density was 1.45 vs 5.20 (difference 3.75, 2.80 to 4.65). Critical+High density was 0.35 vs 0.75 (difference 0.40, −0.10 to 1.00). The pooled quality result was driven principally by dietetics.
Conclusions
AI-generated notes achieved modestly higher pooled quality and fewer transcript-grounded errors, chiefly omissions; serious-error evidence was inconclusive. Small discipline samples and incomplete ground truth preclude safety or discipline-wide superiority claims.
