AI-Enhanced Triage: Improving Musculoskeletal Care with Data Driven Screening of Referral Letters

Background Thirty percent of General Practitioners' (GP) consultations concern musculoskeletal symptoms (MSK). Many of those are referred to a rheumatology outpatient clinic. Triaging the GPs’ referral letters is an important task to prioritise patients in need of early treatment such as those likely to have rheumatoid arthritis. The letters may provide valuable insights into patients' conditions beyond the reasons for referral described by the GPs. Objective We set out to build a Machine Learning (ML) pipeline using only GP referral letters to accelerate prioritisation of referrals by identifying: I) RA, II) osteoarthritis, III) fibromyalgia and IV) patients who remain under the care of a rheumatologist for > 3 months. Methods We collected 8,044 valid referral letters from 5,728 patients across 12 different clinics affiliated to Reumazorg Zuid West Nederland, splitting the data into a development set (Roosendaal and Goes; n = 7,213) and an external replication set (remaining centres; n = 831). We preprocessed the narrative data by I) removing named entities, II) batch correction, III) and transforming the data to a vector with term frequency by inverse document frequency. For each disease we trained an eXtreme Gradient Boosting (XGB) model, which we fine tuned across 1000 epochs with a Bayesian optimization technique, and validated on a hold out data set. Next, we assessed the generalizability to other centres by applying our models on the external data. Finally, we compared the prioritisation ability of our RA-classifier versus the current referral order to assess the additive value of ML to patient triaging. Results The pipeline produced models with a consistent performance in the replication set. Our classifiers did not merely ‘extract’ the suspected diagnosis by the GP, rather it identifies implicit structure in the text indicative of the disease in question. We achieved an AUC-ROC of 0.78 (CI:0.74–0.83) for RA, 0.71 (CI: 0.67–0.74) for osteoarthritis, 0.81 (CI: 0.77–0.85) for fibromyalgia and 0.63 (CI: 0.61–0.66) for chronic follow-up. For RA we could define a cut-off that would preselect two-thirds of all RA patients, while excluding two thirds of non-RA patients, both in holdout and final test set. Finally, our classifier was able to prioritise RA over non-RA cases ( P < 0.001 ), while the manual referral system could not significantly differentiate between RA and non-RA. Conclusion Prior to a rheumatology outpatient clinic visit, we could effectively identify patients with high risk for RA, Osteoarthritis and Fibromyalgia. Patients that were chronically followed-up at the rheumatology clinic were difficult to predict, likely due to patients’ heterogeneity. For all four classification tasks we could improve visit prioritisation compared to current waiting times. Overall, our algorithm has potential to improve care efficiency, reduce clinicians' workload, and facilitate early specialised care. Future research is needed to integrate these models into a comprehensive decision support tool for clinical use.