Development of an AI-enabled predictive model to identify the ‘sick child’ at a pediatric telemedicine and medication delivery service in Haiti
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
One of the most difficult challenges in pediatric telemedicine is to accurately discriminate between the ‘sick’ and ‘not sick’ child, especially in resource-limited settings. Models that flag potentially ‘sick’ cases for additional safety checks represent an opportunity for telemedicine to reach its potential. However, there are critical knowledge gaps on how to develop such models and integrate them into electronic clinical decision support (eCDS) tools.
Methods
To address this challenge, we developed a study design that utilized data from paired virtual and in-person exams at a telemedicine and medication delivery service (TMDS) in Haiti. Providers were allowed to mark respondent data as potentially unreliable. Artificial intelligence /machine learning (XGBoost) was applied to analyze paired data from participants across three consecutive implementation studies. Model derivation focused on identifying ‘sick’ patients (not-mild) and those requiring escalation. An ensemble method, based on gradient boosted decision-trees, was used given the limited sample size. The area under the receiver operating characteristic curve (AUC) was the primary outcome measure.
Results
A total of 683 paired records were available for this secondary analysis from 2225 participants enrolled. The median age was 15 months and 47% were female. For prediction of a ‘sick’ child, we found an AUC of 0.82 (95% CI 0.78-0.86) after 5-fold cross validation; calibration slope and intercept were 1.31 (95%CI:1.09-1.53) and 0.04 (95%CI:-0.14-0.23), respectively. For prediction of escalation, we found an AUC of 0.78 (95%CI:0.74-0.81); calibration slope and intercept were 0.63 (95%CI:0.52-0.74) and 0.05 (95%CI:0.52-0.74), respectively. Accounting for data marked as potentially unreliable had mixed effects.
Interpretation
These methods and findings offer an innovative and important proof-of-concept to improve pediatric telemedicine. The models require external validation prior to eCDS integration and deployment. Once validated, the models are designed to provide a critical safety check for experienced providers and digitally convey expertise to new providers.
Funding
National Institutes of Health (USA) grants to EJN (R21TW012332; DP5OD019893), internal funding at UF (Children’s Miracle Network), and private donations.
RESEARCH IN CONTEXT
Evidence before this study
We conducted two Pubmed searches for reports published in all languages. The first search terms were (telemedicine) AND (artificial intelligence OR machine learning) AND (pediatrics OR paediatrics). The primary search criteria identified 153 publications and reviews were excluded leaving 101 papers all published after 1998. Enumerated results by named subspecialty were neurology (n=2), ophthalmology (n=13), otology (n=3), endocrinology (n=11), cardiology (n=6), pulmonology (n=3), gastroenterology(n=1), dermatology (n=1) and surgery (n=22). Ten of the publications focused on global health or low-middle income countries (LMIC) populations. The second search terms were ((Telemedicine) AND (delivery OR paramedicine) AND (pediatrics OR paediatrics)) AND (global health OR LMIC) which generated 99 publications and 76 papers remained after reviews were excluded, all published after 2015. After manual evaluation of the results from both searches, no publications were identified that fully met the scope of this paper. Examples of telemedicine research for concordance with paired exams does exist 1,2 .
Added value of this study
To the best of our knowledge, this is the first study that investigated pediatric disease severity prediction using virtual and in-person exams in the context of telemedicine -- for either high or low resourced settings. Therefore, the added value of this study is an innovative and important proof-of-concept to improve telemedicine research and clinical practice beyond the scope of global health.
Implications of all the available evidence
Inside the field of global health, there is a need to develop evidence-based approaches to extend care early to pediatric patients who may be isolated by poverty, geography or unrest. This must be done safely and this paper offers an approach to develop and incorporate disease severity prediction models into eCDS tools. In addition, these tools may serve as a welcomed safety check for experienced providers and a method to digitally convey expertise to new providers as these services scale.
