Accuracy of automated computer aided-risk scoring systems to estimate the risk of COVID-19 and in-hospital mortality: a retrospective cohort study

Abstract

Objectives

Although a set of computer-aided risk scoring systems (CARSS), that use the National Early Warning Score and routine blood tests results, have been validated for predicting in-hospital mortality and sepsis in unplanned admission to hospital, little is known about their performance for COVID-19 patients. We compare the performance of CARSS in unplanned admissions with COVID-19 during the first phase of the pandemic.

Design

a retrospective cross-sectional study

Setting

Two acute hospitals (Scarborough and York) are combined into a single dataset and analysed collectively.

Participants

Adult (>=18 years) non-elective admissions discharged between 11-March-2020 to 13-June-2020 with an index NEWS electronically recorded within ±24 hours. We assessed the performance of all four risk score (for sepsis: CARS_N, CARS_NB; for mortality: CARM_N, CARM_NB) according to discrimination (c-statistic) and calibration (graphically) in predicting the risk of COVID-19 and in-hospital mortality.

Results

The risk of in-hospital mortality following emergency medical admission was 8.4% (500/6444) and 9.6% (620/6444) had a diagnosis of COVID-19. For predicting COVID-19 admissions, the CARS_N model had the highest discrimination 0.73 (0.71 to 0.75) and calibration slope 0.81 (0.72 to 0.89). For predicting in-hospital mortality, the CARM_NB model had the highest discrimination 0.84 (0.82 to 0.75) and calibration slope 0.89 (0.81 to 0.98).

Conclusions

Two of the computer-aided risk scores (CARS_N and CARM_NB) are reasonably accurate for predicting the risk of COVID-19 and in-hospital mortality, respectively. They may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned hospital admissions because they are automated and require no additional data collection.

Article Summary

In this study, we found that two of the automated computer-aided risk scores are reasonably accurate for predicting the risk of COVID-19 and in-hospital mortality, respectively.
They may be clinically useful as an early warning system at the time of admission especially to triage large numbers of unplanned hospital admissions because they are automated and require no additional data collection.
Although we focused on in-hospital mortality (because we aimed to aid clinical decision making in the hospital), the impact of this selection bias needs to be assessed by capturing out-of-hospital mortality by linking death certification data and hospital data.
We identified COVID-19 based on ICD-10 code ‘U071’ which was determined by COVID-19 swab test results (hospital or community) and clinical judgment and so our findings are constrained by the accuracy of these methods

SciScore for 10.1101/2020.12.01.20241828: (What is this?)

Please note, not all rigor criteria are appropriate for all manuscripts.

Table 1: Rigor

Institutional Review Board Statement	not detected.
Randomization	We determined the discrimination of CARSS using the concordance statistic (c-statistic) that gives the probability of randomly selected patients who experienced adverse outcome had a higher risk score than a patient who does not.
Blinding	not detected.
Power Analysis	not detected.
Sex as a biological variable	For each emergency admission, we obtained a pseudonymised patient identifier, patient’s age (years), gender (male/female), discharge status (alive/dead), admission and discharge date and time, diagnoses codes based on the 10th revision of the International Statistical Classification of Diseases (ICD-10) [9] [10], NEWS (including its subcomponents respiratory rate, temperature, systolic pressure, pulse rate, oxygen saturation, oxygen supplementation, oxygen scales 1 & 2, and alertness including confusion) [4,11], blood test results (albumin, creatinine, haemoglobin, potassium, sodium, urea, and white cell count), and Acute Kidney Injury (AKI) score.

Table 2: Resources

No key resources detected.

Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:

There are several limitations to our study. (1) This is data from a single NHS Trust and the extent to which these findings are generalisable, further study is required. (2) We used the index NEWS and blood test results which reflects the ‘on-admission’ risk of mortality of the patients. Nonetheless, NEWS and blood test results are repeatedly updated for each patient according to local hospital protocols (Figure S5 in supplementary material). (3) Although we focused on in-hospital mortality (because we aimed to aid clinical decision making in the hospital), the impact of this selection bias needs to be assessed by capturing out-of-hospital mortality by linking death certification data and hospital data. (4) We identified COVID-19 based on ICD-10 code ‘U071’ which was determined by COVID-19 swab test results (hospital or community) and clinical judgment and so our findings are constrained by the accuracy of these methods [24,25].

Results from TrialIdentifier: No clinical trial numbers were referenced.

Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

Results from JetFighter: We did not find any issues relating to colormaps.

Results from rtransparent:

Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
No protocol registration statement was detected.

Read the original source