Development and validation of an Explainable Machine Learning Model for Predicting Multiple Organ Failure in Patients with Acute Pancreatitis: a Multicenter Cohort Study

1.1. Background Acute pancreatitis can lead to a serious and life-threatening situation known as multiple organ failure (MOF). Timely and precise forecasting and detection of MOF are essential. Predictive models utilizing machine learning have shown potential in forecasting MOF in emergency situations. Nevertheless, their use for predicting MOF specifically in patients with acute pancreatitis is still not widespread. This research seeks to create and confirm a machine learning-based model for predicting MOF in individuals suffering from acute pancreatitis. 1.2. Methods This research utilized two retrospective cohorts for the purposes of developing, validating, and testing the model. The derivation and validation cohorts were sourced from the MIMIC-IV database, which was divided randomly into two segments (70% for model development and 30% for internal validation). For external validation, a retrospective cohort from the eICU database was used. After performing several data preprocessing techniques, including interpolation and standardization, we applied four methods for feature selection, ultimately identifying 15 key features. Seven machine learning algorithms were utilized to create predictive models, with their performance assessed through various metrics such as ROC, DCA, PRC, calibration curve, and confusion matrice. The final model's interpretation was carried out using SHAP technology, and a web-based risk calculator was created for use in clinical settings. 1.3. Results The model was created utilizing data from the MIMIC-Ⅳ database (n = 582) and underwent validation and testing with both the MIMIC-Ⅳ (n = 250) and eICU (n = 474) datasets. We identified the best feature combination through four different selection techniques and seven machine learning algorithms. A variable that was recognized by all four selection methods was included in the model development process. The most effective model, CatBoost, was built using 15 easily accessible admission features, resulting in an AUC of 0.939, an F1 score of 0.795, and an accuracy of 0.856 during validation. Subsequently, hyperparameter optimization was conducted on the derivation cohort via 5-fold cross-validation and grid search. The final model was assessed on the test dataset, yielding an AUC of 0.855, an F1 score of 0.658, and an accuracy of 0.761. Additionally, SHAP analysis indicated that INR, Cr, and PO2 are the three most significant variables affecting the model's predictions. This model has been transformed into an online clinical tool to enhance its application in healthcare environments. 1.4. Conclusion The CatBoost model, which is interpretable, effectively forecasted the likelihood of MOF in individuals suffering from acute pancreatitis, showcasing strong predictive performance in both internal and external validation groups. Additionally, this model has been implemented as an online tool for risk assessment to improve its practical application in clinical settings.