Prospective validation of an uncertainty-aware machine learning model for secondary progressive multiple sclerosis classification
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Relapsing–remitting multiple sclerosis (RRMS) often progresses to secondary-progressive MS (SPMS), but diagnosis is typically retrospective, delaying potential interventions. Existing rule-based and machine learning approaches often lack prospective validation and provide no uncertainty estimates, limiting clinical adoption.
Objective
To prospectively evaluate a conformal prediction (CP)-enabled random forest model for reproducing clinician-recorded disease course labels in the Swedish Multiple Sclerosis Registry (SMSReg) and compare its classifications with the Lorscheider criteria.
Methods
The CP model was trained and calibrated on SMSReg-2022 (10,787 individuals, 85,461 hospital visits) and then validated on SMSReg-2025 (15,719 individuals, 45,077 hospital visits; not included in the SMSReg-2022 training/calibration/testing datasets). Performance was assessed at the hospital visit and individual levels and compared with the Lorscheider criteria.
Results
The model maintained good calibration on the prospective dataset. At the per-hospital visit-level, the model accuracies were 92% (RRMS) and 95% (SPMS). At the individual level, sensitivity was 98%, specificity 89%, and F1-score 0.75. With clinician-recorded labels as reference, the model showed higher sensitivity than the Lorscheider criteria (98% vs. 71%), more transition assignments (90% vs. 57%), and higher F1-score (0.76 vs. 0.69).
Conclusion
The CP-enabled model prospectively reproduced clinician-recorded disease course labels with calibrated uncertainty, supporting harmonization and decision support.
