Using Vision and Large Language Models for Psychological Intervention Competency Assessment: A Pilot Study of Agreement with Clinician Ratings

Read the full article See related articles

Listed in

This article is not in any list yet, why not save it to one of your lists.
Log in to save this article

Abstract

Introduction: Supervision in psychological intervention training is limited by the time needed to evaluate simulated sessions. We tested a Vision Language Model (VLM) and Large Language Model (LLM) pipeline that assesses competency from verbal and nonverbal communication. Method: Gemini-3.1-Pro was used to annotate behaviors in 25 simulated therapy videos and incorporate them into transcripts, and ChatGPT-6.1-Sol rated them against the ENACT scale. Two clinicians verified VLM-identified behaviors and four independently rated each video. Agreement was estimated with Gwet's AC1 and ICC. Results: Of 1,558 VLM-identified behaviors, 70% were confirmed by both clinicians and 93% by at least one. AI-clinician agreement on therapeutic behavior attributes (AC1=0.73, 95% CI 0.70-0.76) approached between-clinician agreement (0.75, 0.72-0.78). AI-clinician ICCs on therapeutic domain levels (median 0.57) were modestly lower than between-clinician ICCs (0.67). Discussion: The pipeline agreed with clinicians nearly as well as clinicians agreed with each other and shows promise as a supplementary evaluator in therapist training.

Article activity feed