Development and calibration of a simple mortality risk score for hospitalized COVID-19 adults

  1. Edwin Yoo
  2. Bethany Percha
  3. Max Tomlinson
  4. Victor Razuk
  5. Stephanie Pan
  6. Madeleine Basist
  7. Pranai Tandon
  8. Jing Gennie Wang
  9. Cynthia Gao
  10. Sonali Bose
  11. Umesh Gidwani

Reviewed by

Read the full article

Discuss this preprint

Start a discussion What are Sciety discussions?

Listed in

Log in to save this article

Abstract

Objectives

Mortality risk scores, such as SOFA, qSOFA, and CURB-65, are quick, effective tools for communicating a patient’s prognosis and guiding therapeutic decisions. Most use simple calculations that can be performed by hand. While several COVID-19 specific risk scores exist, they lack the ease of use of these simpler scores. The objectives of this study were (1) to design, validate, and calibrate a simple, easy-to-use mortality risk score for COVID-19 patients and (2) to recalibrate SOFA, qSOFA, and CURB-65 in a hospitalized COVID-19 population.

Design

Retrospective cohort study incorporating demographic, clinical, laboratory, and admissions data from electronic health records.

Setting

Multi-hospital health system in New York City. Five hospitals were included: one quaternary care facility, one tertiary care facility, and three community hospitals.

Participants

Patients (n=4840) with laboratory-confirmed SARS-CoV2 infection who were admitted between March 1 and April 28, 2020.

Main outcome measures

Gray’s K-sample test for the cumulative incidence of a competing risk was used to assess and rank 48 different variables’ associations with mortality. Candidate variables were added to the composite score using DeLong’s test to evaluate their effect on predictive performance (AUC) of in-hospital mortality. Final AUCs for the new score, SOFA, qSOFA, and CURB-65 were assessed on an independent test set.

Results

Of 48 variables investigated, 36 (75%) displayed significant (p<0.05 by Gray’s test) associations with mortality. The variables selected for the final score were (1) oxygen support level, (2) troponin, (3) blood urea nitrogen, (4) lymphocyte percentage, (5) Glasgow Coma Score, and (6) age. The new score, COBALT, outperforms SOFA, qSOFA, and CURB-65 at predicting mortality in this COVID-19 population: AUCs for initial, maximum, and mean COBALT scores were 0.81, 0.91, and 0.92, compared to 0.77, 0.87, and 0.87 for SOFA. We provide COVID-19 specific mortality estimates at all score levels for COBALT, SOFA, qSOFA, and CURB-65.

Conclusions

The COBALT score provides a simple way to estimate mortality risk in hospitalized COVID-19 patients with superior performance to SOFA and other scores currently in widespread use. Evaluation of SOFA, qSOFA, and CURB-65 in this population highlights the importance of recalibrating mortality risk scores when they are used under novel conditions, such as the COVID-19 pandemic. This study’s approach to score design could also be applied in other contexts to create simple, practical and high-performing mortality risk scores.

Trial registration

NA

Funding source

The authors declare that there was no external funding provided.

Summary box

What is already known on this topic

  • Mortality risk scores are widely used in clinical settings to facilitate communication with patients and families, guide goals of care discussions, and optimize resource allocation.

  • Although popular mortality risk scores like SOFA, qSOFA, and CURB-65 are routinely used in COVID-19 populations, they were originally calibrated in different contexts and their true performance among hospitalized COVID-19 patients is unknown.

  • Several dedicated COVID-19 mortality risk scores have been created during the 2019-2020 pandemic, but all use complicated formulae or machine learning algorithms and are difficult or impossible to calculate by hand, limiting their applicability at the bedside.

What this study adds

  • We describe a data-driven, simple, and hand-calculable COVID-specific mortality risk score (COBALT) that has superior performance to SOFA, qSOFA, and CURB-65 in a hospitalized COVID-19 patient population.

  • We provide COVID-specific mortality estimates for SOFA, qSOFA, and CURB-65 using data from 4840 patients in a large and diverse New York City multihospital health system.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2020.08.31.20185363: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: Study population and dataset: This study was approved by the Mount Sinai Institutional Review Board (IRB-20-03613).
    RandomizationWe randomly divided the patients into three groups.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Software and libraries: All preprocessing of the raw electronic health record data was performed in Python (version 3.7.7).
    Python
    suggested: (IPython, RRID:SCR_001658)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    As a retrospective study within a single health system, our study also has some limitations. First, documentation of all kinds was inconsistent during the first wave of COVID-19, and the environments at different hospitals varied substantially. While it is unlikely that a laboratory result or medication administration was missed, inconsistencies in flowsheet documentation during this period could mean that the timings of different modes of oxygen administration were not always accurately captured. Second, the score may need to be calibrated separately when used outside hospital settings. For example, COBALT mortality thresholds reported here may differ when applied to skilled nursing facilities or to health systems in countries outside the U.S. Finally, we were unable to include two health system hospitals (Mount Sinai Beth Israel and Mount Sinai South Nassau) because, as of the time of this study, they did not yet use the Epic electronic health record system. The COBALT score provides an easy-to-use, data-driven bedside tool to assess the risk of mortality in COVID-19 patients and outperforms other hospital mortality risk scores currently in widespread use. In addition, our analysis and recalibration of existing scores is, we believe, unique in the COVID-19 literature. We hope our score and the approach we took to design it will prove useful to our colleagues in the United States and throughout the world who continue to fight the COVID-19 pandemic.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2020.08.31.20185363 on medRxiv