Evaluation of a Machine Learning Approach Utilizing Wearable Data for Prediction of SARS-CoV-2 Infection in Healthcare Workers
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
- Evaluated articles (Rapid Reviews Infectious Diseases)
Abstract
Importance
Passive and non-invasive identification of SARS-CoV-2 infection remains a challenge. Widespread use of wearable devices represents an opportunity to leverage physiological metrics and fill this knowledge gap.
Objective
To determine whether a machine learning model can detect SARS-CoV-2 infection from physiological metrics collected from wearable devices.
Design
A multicenter observational study enrolling health care workers with remote follow-up.
Setting
Seven hospitals from the Mount Sinai Health System in New York City
Participants
Eligibility criteria included health care workers who were ≥18 years, employees of one of the participating hospitals, with at least an iPhone series 6, and willing to wear an Apple Watch Series 4 or higher. We excluded participants with underlying autoimmune/inflammatory diseases, and medications known to interfere with autonomic function. We enrolled participants between April 29 th , 2020, and March 2 nd , 2021, and followed them for a median of 73 days (range, 3-253 days). Participants provided patient-reported outcome measures through a custom smartphone application and wore an Apple Watch, collecting heart rate variability and heart rate data, throughout the follow-up period.
Exposure
Participants were exposed to SARS-CoV-2 infection over time due to ongoing community spread.
Main Outcome and Measure
The primary outcome was SARS-CoV-2 infection, defined as ±7 days from a self-reported positive SARS-CoV-2 nasal PCR test.
Results
We enrolled 407 participants with 49 (12%) having a positive SARS-CoV-2 test during follow-up. We examined five machine-learning approaches and found that gradient-boosting machines (GBM) had the most favorable 10-CV performance. Across all testing sets, our GBM model predicted SARS-CoV-2 infection with an average area under the receiver operating characteristic (auROC)=85% (Confidence Interval 83-88%). The model was calibrated to improve sensitivity over specificity, achieving an average sensitivity of 76% (CI ±∼4%) and specificity of 84% (CI ±∼0.4%). The most important predictors included parameters describing the circadian HRV mean (MESOR) and peak-timing (acrophase), and age.
Conclusions and Relevance
We show that a tree-based ML algorithm applied to physiological metrics passively collected from a wearable device can identify and predict SARS-CoV2 infection. Utilizing physiological metrics from wearable devices may improve screening methods and infection tracking.
Article activity feed
-
Aaron Hudson
Review 1: "Evaluation of a Machine Learning Approach Utilizing Wearable Data for Prediction of SARS-CoV-2 Infection in Healthcare Workers"
This study develops a prediction model for positive COVID-19 diagnosis using data collected from Apple Watches on heart rate variability (HRV) among healthcare workers. Reviewers highlight unclear model justifications and methodology.
-
Toyya Pujol
Review 2: "Evaluation of a Machine Learning Approach Utilizing Wearable Data for Prediction of SARS-CoV-2 Infection in Healthcare Workers"
This study develops a prediction model for positive COVID-19 diagnosis using data collected from Apple Watches on heart rate variability (HRV) among healthcare workers. Reviewers highlight unclear model justifications and methodology.
-
Strength of evidence
Reviewers: A Hudson (UC Berkeley) | 📒📒📒 ◻️◻️
T Pujol (RAND Corporation) | 📒📒📒 ◻️◻️ -
SciScore for 10.1101/2021.11.04.21265931: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Antibodies Sentences Resources Subjects completed daily surveys to report any COVID-19 related symptoms, symptom severity, the results for any SARS-CoV-2 nasal PCR tests, and SARS-CoV-2 antibody test results. SARS-CoV-2suggested: NoneResults from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are several limitations to our study. First, HRV was collected sporadically by the Apple Watch. We employed statistical modeling …
SciScore for 10.1101/2021.11.04.21265931: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
Antibodies Sentences Resources Subjects completed daily surveys to report any COVID-19 related symptoms, symptom severity, the results for any SARS-CoV-2 nasal PCR tests, and SARS-CoV-2 antibody test results. SARS-CoV-2suggested: NoneResults from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are several limitations to our study. First, HRV was collected sporadically by the Apple Watch. We employed statistical modeling to account for this. However, a denser data set using continuous data would likely further improve our predictions. Second, the model we employed used a 7-day smoothing approach. This approach observed infection-induced changes in HRV later than if HRV was estimated using a single-day method. Thus, the approach we employed is conservative. An additional limitation is that the Apple Watch provides HRV measurements only in the SDDN time domain. This limits assessments between other types of HRV measurements and COVID-19 outcomes. Additionally, other factors might impact HRV, which we were not able to capture and control for in the analysis. Furthermore, we were not routinely checking for SARs-CoV-2 infections and relied on subjects reporting a COVID-19 diagnosis. Therefore, infections could have occurred that are not accounted. Lastly, we did not externally validate our machine learning algorithm in another cohort.
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
Results from scite Reference Check: We found no unreliable references.
-