Development and validation of an algorithm to estimate the risk of severe complications of COVID-19: a retrospective cohort study in primary care in the Netherlands

  1. Ron M C Herings
  2. Karin M A Swart
  3. Bernard A M van der Zeijst
  4. Amber A van der Heijden
  5. Koos van der Velden
  6. Eric G Hiddink
  7. Martijn W Heymans
  8. Reinier A R Herings
  9. Hein P J van Hout
  10. Joline W J Beulens
  11. Giel Nijpels
  12. Petra J M Elders

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

To develop an algorithm (sCOVID) to predict the risk of severe complications of COVID-19 in a community-dwelling population to optimise vaccination scenarios.

Design

Population-based cohort study.

Setting

264 Dutch general practices contributing to the NL-COVID database.

Participants

6074 people aged 0–99 diagnosed with COVID-19.

Main outcomes

Severe complications (hospitalisation, institutionalisation, death). The algorithm was developed from a training data set comprising 70% of the patients and validated in the remaining 30%. Potential predictor variables included age, sex, chronic comorbidity score (CCS) based on risk factors for COVID-19 complications, obesity, neighbourhood deprivation score (NDS), first or second COVID-19 wave and confirmation test. Six population vaccination scenarios were explored: (1) random ( naive ), (2) random for persons above 60 years ( 60plus ), (3) oldest patients first in age band of 5 years ( oldest first ), (4) target population of the annual influenza vaccination programme ( influenza ), (5) those 25–65 years of age first ( worker ), and (6) risk based using the prediction algorithm ( sCOVID ).

Results

Severe complications were reported in 243 (4.8%) people with 59 (20.3%) nursing home admissions, 181 (62.2%) hospitalisations and 51 (17.5%) deaths. The algorithm included age, sex, CCS, NDS, wave and confirmation test (c-statistic=0.91, 95% CI 0.88 to 0.94) in the validation set. Applied to different vaccination scenarios, the proportion of people needed to be vaccinated to reach a 50% reduction of severe complications was 67.5%, 50.0%, 26.1%, 16.0%, 10.0% and 8.4% for the worker, naive , influenza, 60plus, oldest first and sCOVID scenarios, respectively.

Conclusion

The sCOVID algorithm performed well to predict the risk of severe complications of COVID-19 in the first and second waves of COVID-19 infections in this Dutch population. The regression estimates can and need to be adjusted for future predictions. The algorithm can be applied to identify persons with highest risks from data in the electronic health records of general practitioners (GPs).

Article activity feed

  1. Version published to 10.1136/bmjopen-2021-050059
  2. ScreenIT

    SciScore for 10.1101/2021.02.05.21251197: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    NIH rigor criteria are not applicable to paper type.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    GLMNET package (4.0-2), version was used for statistical analyses and constructing figures.
    GLMNET
    suggested: (glmnet, RRID:SCR_015505)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Strengths and limitations: The NL-COVID database also has limitations and strengths. First, we have substantial underreporting of positive cases since our 264 registration practices consisting of about 5% of all GP practices only reported 0.7% of the registered cases. This is explained by several factors: first, practices enrolled into the program over time and some practices only joined the program and the end of 2021. Second, COVID-19 testing was done by the regional health authorities (GGD) whereas the administrations of the GGD were not linked with the GP administration. Therefore our registration relies on whether the patient contacted the GP and whether the GP registered the patient. This makes it likely that we have a selection bias towards the more severe disease manifestations of the COVID-19 infection. Also our prediction partly relied on the judgement of the GP whether a patient was COVID 19 positive (in case of lacking test results). It should therefore be stressed that absolute estimates risk estimates to develop severe complications should be interpreted with care only by health care professionals for prioritizing strategies. A weakness of the sCOVID scenario is that we did not perform an external validation. The large number of GP practices that came from all over the country and the good testing characteristics of the validation set, makes it likely that the accuracy of the scenarios is adequate. For the comparison of the different scenarios this has no import...

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from scite Reference Check: We found one citation with an erratum. We recommend checking the erratum to confirm that it does not impact the accuracy of your citation.

    DOIStatusTitle
    10.1161/circoutcomes.113.000152Has correctionDevelopment and Validation of a Risk Score to Predict QT Int…

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2021.02.05.21251197v1 on medRxiv