Internal Validation of a Machine Learning Model for Antimicrobial Stewardship: Evaluating Trainability of Data and the Accuracy of Clinical Recommendations Within a Clinical Decision Support System

Background: Antimicrobial stewardship programs (ASPs) are essential in combating antimicrobial resistance (AMR), however, limited resources hinder their implementation. Arkstone, a biotechnology company, developed a machine learning (ML)-driven clinical decision support system (CDSS) to guide antimicrobial prescribing. Though widely used, the model had not been previously evaluated. Methods: Three components of the ML system were assessed: (1) A prospective observational study tested its ability to distinguish trained from novel data using various validation techniques and BioFire molecular panel inputs; (2) An anonymous retrospective analysis of internal infectious disease lab results evaluated recognition of novel versus trained complex datasets; and (3) A prospective observational validation study reviewed clinical recommendations against standard guidelines by independent clinicians. Results: The system achieved 100% accuracy (F1=1) in identifying 111 unique novel data points across 1,110 tests over nine training sessions. It correctly identified all 519 fully trained and 644 novel complex datasets. Among 644 clinician-trained reports, there were no major discrepancies in recommendations, with only 100 showing minor differences. Conclusion: This novel ML system demonstrated high accuracy in distinguishing trained from novel data and produced recommendations consistent with clinical guidelines. These results support its value in strengthening CDSS and ASP efforts.