Machine Learning to Predict Mortality and Critical Events in a Cohort of Patients With COVID-19 in New York City: Model Development and Validation

  1. Akhil Vaid
  2. Sulaiman Somani
  3. Adam J Russak
  4. Jessica K De Freitas
  5. Fayzan F Chaudhry
  6. Ishan Paranjpe
  7. Kipp W Johnson
  8. Samuel J Lee
  9. Riccardo Miotto
  10. Felix Richter
  11. Shan Zhao
  12. Noam D Beckmann
  13. Nidhi Naik
  14. Arash Kia
  15. Prem Timsina
  16. Anuradha Lala
  17. Manish Paranjpe
  18. Eddye Golden
  19. Matteo Danieletto
  20. Manbir Singh
  21. Dara Meyer
  22. Paul F O'Reilly
  23. Laura Huckins
  24. Patricia Kovatch
  25. Joseph Finkelstein
  26. Robert M. Freeman
  27. Edgar Argulian
  28. Andrew Kasarskis
  29. Bethany Percha
  30. Judith A Aberg
  31. Emilia Bagiella
  32. Carol R Horowitz
  33. Barbara Murphy
  34. Eric J Nestler
  35. Eric E Schadt
  36. Judy H Cho
  37. Carlos Cordon-Cardo
  38. Valentin Fuster
  39. Dennis S Charney
  40. David L Reich
  41. Erwin P Bottinger
  42. Matthew A Levin
  43. Jagat Narula
  44. Zahi A Fayad
  45. Allan C Just
  46. Alexander W Charney
  47. Girish N Nadkarni
  48. Benjamin S Glicksberg

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

COVID-19 has infected millions of people worldwide and is responsible for several hundred thousand fatalities. The COVID-19 pandemic has necessitated thoughtful resource allocation and early identification of high-risk patients. However, effective methods to meet these needs are lacking.

Objective

The aims of this study were to analyze the electronic health records (EHRs) of patients who tested positive for COVID-19 and were admitted to hospitals in the Mount Sinai Health System in New York City; to develop machine learning models for making predictions about the hospital course of the patients over clinically meaningful time horizons based on patient characteristics at admission; and to assess the performance of these models at multiple hospitals and time points.

Methods

We used Extreme Gradient Boosting (XGBoost) and baseline comparator models to predict in-hospital mortality and critical events at time windows of 3, 5, 7, and 10 days from admission. Our study population included harmonized EHR data from five hospitals in New York City for 4098 COVID-19–positive patients admitted from March 15 to May 22, 2020. The models were first trained on patients from a single hospital (n=1514) before or on May 1, externally validated on patients from four other hospitals (n=2201) before or on May 1, and prospectively validated on all patients after May 1 (n=383). Finally, we established model interpretability to identify and rank variables that drive model predictions.

Results

Upon cross-validation, the XGBoost classifier outperformed baseline models, with an area under the receiver operating characteristic curve (AUC-ROC) for mortality of 0.89 at 3 days, 0.85 at 5 and 7 days, and 0.84 at 10 days. XGBoost also performed well for critical event prediction, with an AUC-ROC of 0.80 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. In external validation, XGBoost achieved an AUC-ROC of 0.88 at 3 days, 0.86 at 5 days, 0.86 at 7 days, and 0.84 at 10 days for mortality prediction. Similarly, the unimputed XGBoost model achieved an AUC-ROC of 0.78 at 3 days, 0.79 at 5 days, 0.80 at 7 days, and 0.81 at 10 days. Trends in performance on prospective validation sets were similar. At 7 days, acute kidney injury on admission, elevated LDH, tachypnea, and hyperglycemia were the strongest drivers of critical event prediction, while higher age, anion gap, and C-reactive protein were the strongest drivers of mortality prediction.

Conclusions

We externally and prospectively trained and validated machine learning models for mortality and critical events for patients with COVID-19 at different time horizons. These models identified at-risk patients and uncovered underlying relationships that predicted outcomes.

Article activity feed

  1. Version published to 10.2196/24018
  2. Version published to 10.2196/preprints.24018
  3. ScreenIT

    SciScore for 10.1101/2020.04.26.20073411: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: This study has been approved by the Institutional Review Board at the Icahn School of Medicine at Mount Sinai (IRB-20-03271)
    RandomizationFor each fold, hyperparameter tuning was performed by randomized grid searching directed towards maximizing the sensitivity metric over 2,000 discrete grid options.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    The results of our models should be considered in light of several limitations. First, we base our predictions solely on a patient’s admission labs (i.e. within 36 hours); while this restriction encourages the use of this model in patient triage, events during a patient’s hospital stay after admission may drive their clinical course away from the prior probability. Furthermore, not all patient labs are drawn at admission, which introduces an element of missingness in our dataset. For example, unlike the general patient population, patients on anticoagulation therapy, who likely have comorbidities increasing their baseline risk, will have coagulation labs (PT, PTT) taken on admission. However, the shift away from predicting death by the model in the absence of PT/PTT (Figure 3) suggests that missingness in coagulation labs is a proxy for this lower baseline risk secondary to not having comorbid conditions that require anticoagulation therapy. Additionally, patients admitted to the hospital later in the crisis were both beneficiaries of improved patient care protocols from experiential learning, but also victims of resource constraints from overburdened hospitals. These effects, while possibly negated by our large sample size, may also induce temporal variation between patient outcomes. Furthermore, inherent limitations exist when using EHRs, especially those integrated from multiple hospitals. In order to facilitate timely dissemination of our results, we chose not to manually...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  4. Version published to 10.1101/2020.04.26.20073411v1 on medRxiv