Comparison of methods for predicting COVID-19-related death in the general population using the OpenSAFELY platform

  1. The OpenSAFELY Collaborative
  2. Elizabeth J. Williamson
  3. John Tazare
  4. Krishnan Bhaskaran
  5. Helen I. McDonald
  6. Alex J. Walker
  7. Laurie Tomlinson
  8. Kevin Wing
  9. Sebastian Bacon
  10. Chris Bates
  11. Helen J. Curtis
  12. Harriet J. Forbes
  13. Caroline Minassian
  14. Caroline E. Morton
  15. Emily Nightingale
  16. Amir Mehrkar
  17. David Evans
  18. Brian D. Nicholson
  19. David A. Leon
  20. Peter Inglesby
  21. Brian MacKenna
  22. Nicholas G. Davies
  23. Nicholas J. DeVito
  24. Henry Drysdale
  25. Jonathan Cockburn
  26. William J. Hulme
  27. Jessica Morley
  28. Ian Douglas
  29. Christopher T. Rentsch
  30. Rohini Mathur
  31. Angel Wong
  32. Anna Schultze
  33. Richard Croker
  34. John Parry
  35. Frank Hester
  36. Sam Harper
  37. Richard Grieve
  38. David A. Harrison
  39. Ewout W. Steyerberg
  40. Rosalind M. Eggo
  41. Karla Diaz-Ordaz
  42. Ruth Keogh
  43. Stephen J. W. Evans
  44. Liam Smeeth
  45. Ben Goldacre

Reviewed by

Read the full article See related articles

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

Background

Obtaining accurate estimates of the risk of COVID-19-related death in the general population is challenging in the context of changing levels of circulating infection.

Methods

We propose a modelling approach to predict 28-day COVID-19-related death which explicitly accounts for COVID-19 infection prevalence using a series of sub-studies from new landmark times incorporating time-updating proxy measures of COVID-19 infection prevalence. This was compared with an approach ignoring infection prevalence.

The target population was adults registered at a general practice in England in March 2020. The outcome was 28-day COVID-19-related death. Predictors included demographic characteristics and comorbidities. Three proxies of local infection prevalence were used: model-based estimates, rate of COVID-19-related attendances in emergency care, and rate of suspected COVID-19 cases in primary care.

We used data within the TPP SystmOne electronic health record system linked to Office for National Statistics mortality data, using the OpenSAFELY platform, working on behalf of NHS England.

Prediction models were developed in case-cohort samples with a 100-day follow-up. Validation was undertaken in 28-day cohorts from the target population. We considered predictive performance (discrimination and calibration) in geographical and temporal subsets of data not used in developing the risk prediction models. Simple models were contrasted to models including a full range of predictors.

Results

Prediction models were developed on 11,972,947 individuals, of whom 7999 experienced COVID-19-related death. All models discriminated well between individuals who did and did not experience the outcome, including simple models adjusting only for basic demographics and number of comorbidities: C-statistics 0.92–0.94. However, absolute risk estimates were substantially miscalibrated when infection prevalence was not explicitly modelled.

Conclusions

Our proposed models allow absolute risk estimation in the context of changing infection prevalence but predictive performance is sensitive to the proxy for infection prevalence. Simple models can provide excellent discrimination and may simplify implementation of risk prediction tools.

Article activity feed

  1. Version published to 10.1186/s41512-022-00120-2
  2. ScreenIT

    SciScore for 10.1101/2021.02.25.21252433: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Results from these models are not shown: the Royston-Parmar models had very similar performance to the Cox models but the Weibull and Gamma models generally had poorer calibration.
    Gamma
    suggested: (GAMMA, RRID:SCR_009484)
    Software and reproducibility: Data management was performed using Python and Google BigQuery, with analysis carried out using Stata 16.1 / Python.
    Python
    suggested: (IPython, RRID:SCR_001658)
    Google BigQuery
    suggested: None

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    This study also has important weaknesses. First, the COVID-19 pandemic is evolving rapidly. We will re-validate and, if necessary, re-calibrate our models using data from the more recent wave, particularly in light of evidence suggesting differences in mortality with the new variant. Further, as the vaccines are rolled out, risk in the population will additionally depend on proportions of different patient groups vaccinated; this could be incorporated into subsequent iterations of these models. We have not yet externally validated these models. However, our internal-external validation suggests little over-optimism in the measures of model performance. Electronic health record data are not collected for research, so information on certain characteristics can be incomplete or absent. For example, ascertainment of HIV and cystic fibrosis is likely to be poor and pregnancy cannot easily be identified, limiting the ability to distinguish risks between these groups. Our approach to missing data reflected the way in which these models might be used in practice if applied within electronic health record systems. For example, patients with no BMI measurement would be assumed to have normal BMI. Our measures of model performance reflect performance under this implementation. The exception is ethnicity – given the strong relationships previously observed between ethnicity and COVID-19 outcomes we chose to restrict our sample to those with recorded ethnicity. In previous work, model est...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.02.25.21252433 on medRxiv