Conformal Learning for Quantifying Uncertainty in Psychiatric Condition Predictions

Purpose The combination of physiological data from ubiquitous fitness trackers like Fitbit® and machine learning presents a promising avenue for improving diagnostic consistency in the psychiatric space. However, machine learning models that utilize large health datasets are hobbled by inconsistent labeling of disorders, a lack of informal treatment recording, variance in individual symptom expressions, and differential representation of sub-groups. We propose utilizing conformal prediction, which quantifies uncertainty by creating prediction sets instead of singleton predictions, as a means of harnessing these promising resources in a reliable and actionable way. Methods This study leverages the National Institutes of Health sponsored All of Us dataset, which included multi-signal Fitbit data from 13,835 participants at the time of the study, to inform machine learning models aimed at enhancing psychiatric diagnosis. We incorporate heart rate, step count, and sleep measurement data to differentiate between subjects with and without an anxiety disorder or depressive disorder. Results We determined that a long short-term memory (LSTM) model trained on the All of Us Fitbit®-derived data, is ultimately, only marginally useful in predicting whether someone has an anxiety or depressive disorder, with a mean accuracy of ~ 64%. However, we were able to effectively use conformal prediction to create prediction sets that contained the true label with 90% confidence. Conclusions We demonstrate the efficacy of conformal methods in producing more reliable and actionable predictions in the psychiatric healthcare space. Our findings suggest that robust machine learning techniques that provide prediction confidence metrics coupled with biometric data could lead to more objective and scalable diagnostic processes, thereby improving patient outcomes.