Artificial Intelligence-based Action Recognition and Skill Assessment in Robotic Cardiac Surgery Simulation

Purpose: To create a deep neural network capable of recognizing basic surgical actions and categorizing surgeons based on their skills using video data only. Materials and Methods: Nineteen surgeons with varying levels of robotic experience performed three wet lab tasks on a porcine model: robotic-assisted atrial closure, mitral stitches, and dissection of the thoracic artery. We used temporal labeling to mark two surgical actions: suturing and dissection. Each complete recording was annotated as either "novice" or "expert" based on the operator's experience. The network architecture combined a Convolutional Neural Network for extracting spatial features with a Long Short-Term Memory layer to incorporate temporal information. Results: A total of 435 recordings were analyzed. The five-fold cross-validation yielded a mean accuracy of 98% for the action recognition (AR) and 79% for the skill assessment (SA) network. The AR model achieved an accuracy of 93%, with average recall, precision, and F1-score all at 93%. The SA network had an accuracy of 56% and a predictive certainty of 95%. Gradient-weighted Class Activation Mapping revealed that the algorithm focused on the needle, suture, and instrument tips during suturing, and on the tissue during dissection. Conclusions: AR network demonstrated high accuracy and predictive certainty, even with a limited dataset. The SA network requires more data to become a valuable tool for performance evaluation. When combined, these deep learning models can serve as a foundation for AI-based automated assessments in robotic cardiac surgery. Public trial registry: ClinicalTrials.gov (NCT05043064)