Empirical Assessment of U.S. Coronavirus Disease 2019 Crisis Standards of Care Guidelines

  1. Maheetha Bharadwaj
  2. Julia L. Jezmir
  3. Sandeep P. Kishore
  4. Marisa Winkler
  5. Bradford Diephus
  6. Hibah Haider
  7. Conor P. Crowley
  8. Mayra Pinilla-Vera
  9. Jack Varon
  10. Rebecca M. Baron
  11. William B. Feldman
  12. Edy Y. Kim

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

To establish the feasibility of empirically testing crisis standards of care guidelines.

DESIGN:

Retrospective single-center study.

SETTING:

ICUs at a large academic medical center in the United States.

SUBJECTS:

Adult, critically ill patients admitted to ICU, with 27 patients admitted for acute respiratory failure due to coronavirus disease 2019 and 37 patients admitted for diagnoses other than coronavirus disease 2019.

INTERVENTIONS:

Review of electronic health record.

MEASUREMENTS AND MAIN RESULTS:

Many U.S. states released crisis standards of care guidelines with algorithms to allocate scarce healthcare resources during the coronavirus disease 2019 pandemic. We compared state guidelines that represent different approaches to incorporating disease severity and comorbidities: New York, Maryland, Pennsylvania, and Colorado. Following each algorithm, we calculated priority scores at the time of ICU admission for a cohort of patients with primary diagnoses of coronavirus disease 2019 and diseases other than coronavirus disease 2019 (n = 64). We assessed discrimination of 28-day mortality by area under the receiver operating characteristic curve. We simulated real-time decision-making by applying the triage algorithms to groups of two, five, or 10 patients. For prediction of 28-day mortality by priority scores, area under the receiver operating characteristic curve was 0.56, 0.49, 0.53, 0.66, and 0.69 for New York, Maryland, Pennsylvania, Colorado, and raw Sequential Organ Failure Assessment score algorithms, respectively. For groups of five patients, the percentage of decisions made without deferring to a lottery were 1%, 57%, 80%, 88%, and 95% for New York, Maryland, Pennsylvania, Colorado, and raw Sequential Organ Failure Assessment score algorithms, respectively. The percentage of decisions made without lottery was higher in the subcohort without coronavirus disease 2019, compared with the subcohort with coronavirus disease 2019.

CONCLUSIONS:

Inclusion of comorbidities does not consistently improve an algorithm’s performance in predicting 28-day mortality. Crisis standards of care algorithms result in a substantial percentage of tied priority scores. Crisis standards of care algorithms operate differently in cohorts with and without coronavirus disease 2019. This proof-of-principle study demonstrates the feasibility and importance of empirical testing of crisis standards of care guidelines to understand whether they meet their goals.

Article activity feed

  1. Version published to 10.1097/cce.0000000000000496
  2. ScreenIT

    SciScore for 10.1101/2020.05.16.20098657: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    Institutional Review Board StatementIRB: Approval was obtained from the Partners HealthCare Institutional Review Board.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.
    Sex as a biological variablenot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Statistical tests were performed in IBM SPSS Statistics Version 25.0.
    SPSS
    suggested: (SPSS, RRID:SCR_002865)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    However, we have identified important limitations of several representative state triage algorithms and provided a framework for future quantitative analyses. The ethical defensibility of these algorithms will depend, in part, on empirical analyses of how they function in practice.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  3. Version published to 10.1101/2020.05.16.20098657v1 on medRxiv