Performance of Machine Learning Models to Predict the Need for Computed Tomography Brain Scans From Clinical Features in Emergency Department Headache Patients

Background There is no unifying diagnostic tool to guide neuroimaging decisions for unselected emergency department (ED) patients presenting with headache. This study aimed to compare the performance characteristics of various machine learning models in predicting the presence of clinically important computed tomography (CT) brain scan findings, thereby determining the need for neuroimaging. Methods Data were combined from our two cohorts of patients with headache who presented to the ED across 10 countries (Australia, New Zealand, Singapore, Hong Kong, United Kingdom, France, Belgium, Romania, Turkey, Israel) during a one-month period in 2019, and in Colombia throughout the 2019 calendar year. Patients aged ≥ 18 years with headache as their chief complaint were included. The primary outcome measure was the presence of clinically important CT findings, including hemorrhages, vascular abnormalities and neoplasm. Patients who did not undergo a CT scan were assumed to have no important findings. Predictor variables included headache features, past medical history, physical examination, and pathology results. Three machine learning models were compared: multivariable logistic regression, elastic net and random forest. The data were split into a training and test set in a 50:50 ratio for modeling. Results Of the 5,290 patients, 2,067 (39.1%) underwent CT scans. A clinically important radiological finding was recorded in 207 (10.0%) of these scans. The logistic regression model showed a slightly higher AUPRC (0.314) than the elastic net (0.295) and random forest models (0.276). It also demonstrated a slightly higher AUROC (0.840) compared with the elastic net (0.834) and random forest (0.797). For the logistic regression model, observed probabilities agreed with predicted probabilities below 0.2, but the model overpredicted when probabilities exceeded 0.2 (slope of calibration curve = 0.71, intercept = − 0.10). Overall, the elastic net model underpredicted, and the random forest model overpredicted. Conclusions Machine learning models can discriminate between the presence and absence of clinically important CT brain scan findings using clinical features of unselected ED patients with headache. The logistic regression model had similar discrimination, but superior calibration compared to the elastic net and random forest models. Machine learning models can potentially help clinicians decide on the need for neuroimaging.