Seek COVER: using a disease proxy to rapidly develop and validate a personalized risk calculator for COVID-19 outcomes in an international network

  1. Ross D. Williams
  2. Aniek F. Markus
  3. Cynthia Yang
  4. Talita Duarte-Salles
  5. Scott L. DuVall
  6. Thomas Falconer
  7. Jitendra Jonnagaddala
  8. Chungsoo Kim
  9. Yeunsook Rho
  10. Andrew E. Williams
  11. Amanda Alberga Machado
  12. Min Ho An
  13. María Aragón
  14. Carlos Areia
  15. Edward Burn
  16. Young Hwa Choi
  17. Iannis Drakos
  18. Maria Tereza Fernandes Abrahão
  19. Sergio Fernández-Bertolín
  20. George Hripcsak
  21. Benjamin Skov Kaas-Hansen
  22. Prasanna L. Kandukuri
  23. Jan A. Kors
  24. Kristin Kostka
  25. Siaw-Teng Liaw
  26. Kristine E. Lynch
  27. Gerardo Machnicki
  28. Michael E. Matheny
  29. Daniel Morales
  30. Fredrik Nyberg
  31. Rae Woong Park
  32. Albert Prats-Uribe
  33. Nicole Pratt
  34. Gowtham Rao
  35. Christian G. Reich
  36. Marcela Rivera
  37. Tom Seinen
  38. Azza Shoaibi
  39. Matthew E. Spotnitz
  40. Ewout W. Steyerberg
  41. Marc A. Suchard
  42. Seng Chan You
  43. Lin Zhang
  44. Lili Zhou
  45. Patrick B. Ryan
  46. Daniel Prieto-Alhambra
  47. Jenna M. Reps
  48. Peter R. Rijnbeek

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

Background

We investigated whether we could use influenza data to develop prediction models for COVID-19 to increase the speed at which prediction models can reliably be developed and validated early in a pandemic. We developed COVID-19 Estimated Risk (COVER) scores that quantify a patient’s risk of hospital admission with pneumonia (COVER-H), hospitalization with pneumonia requiring intensive services or death (COVER-I), or fatality (COVER-F) in the 30-days following COVID-19 diagnosis using historical data from patients with influenza or flu-like symptoms and tested this in COVID-19 patients.

Methods

We analyzed a federated network of electronic medical records and administrative claims data from 14 data sources and 6 countries containing data collected on or before 4/27/2020. We used a 2-step process to develop 3 scores using historical data from patients with influenza or flu-like symptoms any time prior to 2020. The first step was to create a data-driven model using LASSO regularized logistic regression, the covariates of which were used to develop aggregate covariates for the second step where the COVER scores were developed using a smaller set of features. These 3 COVER scores were then externally validated on patients with 1) influenza or flu-like symptoms and 2) confirmed or suspected COVID-19 diagnosis across 5 databases from South Korea, Spain, and the United States. Outcomes included i) hospitalization with pneumonia, ii) hospitalization with pneumonia requiring intensive services or death, and iii) death in the 30 days after index date.

Results

Overall, 44,507 COVID-19 patients were included for model validation. We identified 7 predictors (history of cancer, chronic obstructive pulmonary disease, diabetes, heart disease, hypertension, hyperlipidemia, kidney disease) which combined with age and sex discriminated which patients would experience any of our three outcomes. The models achieved good performance in influenza and COVID-19 cohorts. For COVID-19 the AUC ranges were, COVER-H: 0.69–0.81, COVER-I: 0.73–0.91, and COVER-F: 0.72–0.90. Calibration varied across the validations with some of the COVID-19 validations being less well calibrated than the influenza validations.

Conclusions

This research demonstrated the utility of using a proxy disease to develop a prediction model. The 3 COVER models with 9-predictors that were developed using influenza data perform well for COVID-19 patients for predicting hospitalization, intensive services, and fatality. The scores showed good discriminatory performance which transferred well to the COVID-19 population. There was some miscalibration in the COVID-19 validations, which is potentially due to the difference in symptom severity between the two diseases. A possible solution for this is to recalibrate the models in each location before use.

Article activity feed

  1. Version published to 10.1186/s12874-022-01505-z
  2. Version published to 10.1101/2020.05.26.20112649v4 on medRxiv
  3. Version published to 10.1101/2020.05.26.20112649v3 on medRxiv
  4. ScreenIT

    SciScore for 10.1101/2020.05.26.20112649: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: Consent to publish: All databases obtained institutional review board (IRB) approval or used deidentified data that was considered exempt from IRB approval.
    Consent: Informed consent was not necessary at any site.
    RandomizationThe AUC indicates the probability that for two randomly selected patients, the patient who gets the outcome will be assigned a higher risk.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    No key resources detected.

    Results from OddPub: Thank you for sharing your code and data.

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Limitations: Limitations of the study included being unable to develop a model on COVID-19 patient data due to the scarcity of databases that contain this information in sufficient numbers, however we were able to validate the models developed in COVID-19 and as such are confident the performance is transportable. In CUIMC, HIRA, SIDIAP, and VA COVID-19 databases we either reached or approached the threshold for reliable external validation of ∼100 patients who experience the outcome of interest30,31. The results of TRDW are promising, but might not be reliable due to the low number of outcomes. As larger COVID-19 databases become available, training a model using these data may highlight predictors of severity amongst uncommon influenza presentations, for example younger and healthier patients experiencing severe or critical illness. The calibration in some of the COVID-19 validations could benefit from recalibration which can be performed by either recalibration in the large32 or logistic recalibration33. This suggests that calibration can be an issue in some locations and as such ideally the models will be tested and recalibrated in these locations before use. Further limitations include misclassification of predictors, for example if disease is incorrectly recorded in a patient’s history, as well as in the cohorts through incorrect recording of influenza or COVID-19. We were unable to validate the COVER-H score in CUIMC as it mostly contained ER or hospitalized COVID-19 p...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  5. Version published to 10.1101/2020.05.26.20112649v2 on medRxiv
  6. Version published to 10.1101/2020.05.26.20112649v1 on medRxiv