Profiling and Predicting Faculty Assessment Behavior in Surgical Education
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To study and profile the digital assessment behaviors of surgical faculty and residents, and to build a classifier to predict assessment completion, enhancing formative feedback initiatives.
Background
As competency-based paradigms are integrated into surgical training, developing digital education tools for measuring competency and providing rapid feedback is crucial. Simply making assessments available is inadequate and results in disappointingly low user participation. To optimize engagement and efficacy of these tools, user assessment behaviors need to be studied.
Methods
User data was aggregated from a HIPAA-compliant electronic health record (EHR)-integrated medical education platform. Faculty and resident behaviors were analyzed with respect to factors, such as time, day, device type, automated reminders, and EHR integration. A graphical convolutional neural network (GCN) model was trained to predict faculty participation in completing assessments.
Results
10,729 assessments were completed by 254 attendings for 428 residents across 22 institutions, from 2022 to 2024. 86% of assessments were completed by faculty on weekdays, were significantly influenced by automated platform triggers and EHR integration, and distinct faculty behavior profiles contingent upon time to completion and comment length were established. Residents opened assessments at a median of 1.5 hours of faculty assessment completion, with 96% of assessments viewed by 24 hours. The GCN model successfully predicted faculty assessment completion with 93.5% accuracy and an area under the ROC curve of 0.97.
Conclusions
Faculty assessment behaviors represent an actionable bottleneck which can be leveraged to optimize and tailor the design of digital education tools, to enhance formative feedback.
