Development and validation of automated computer-aided risk scores to predict in-hospital mortality for emergency medical admissions with COVID-19: a retrospective cohort development and validation study
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
There are no established mortality risk equations specifically for unplanned emergency medical admissions which include patients with SARS-19 (COVID-19). We aim to develop and validate a computer-aided risk score (CARMc19) for predicting mortality risk by combining COVID-19 status, the first electronically recorded blood test results and the National Early Warning Score (NEWS2).
Design
Logistic regression model development and validation study.
Setting
Two acute hospitals (York Hospital—model development data; Scarborough Hospital—external validation data).
Participants
Adult (aged ≥16 years) medical admissions discharged over a 24-month period with electronic NEWS and blood test results recorded on admission. We used logistic regression modelling to predict the risk of in-hospital mortality using two models: (1) CARMc19_N: age+sex+NEWS2 including subcomponents+COVID19; (2) CARMc19_NB: CARMc19_N in conjunction with seven blood test results and acute kidney injury score. Model performance was evaluated according to discrimination (c-statistic), calibration (graphically) and clinical usefulness at NEWS2 thresholds of 4+, 5+, 6+.
Results
The risk of in-hospital mortality following emergency medical admission was similar in development and validation datasets (8.4% vs 8.2%). The c-statistics for predicting mortality for CARMc19_NB is better than CARMc19_N in the validation dataset (CARMc19_NB=0.88 (95% CI 0.86 to 0.90) vs CARMc19_N=0.86 (95% CI 0.83 to 0.88)). Both models had good calibration (CARMc19_NB=1.01 (95% CI 0.88 to 1.14) and CARMc19_N:0.95 (95% CI 0.83 to 1.06)). At all NEWS2 thresholds (4+, 5+, 6+) model, CARMc19_NB had better sensitivity and similar specificity.
Conclusions
We have developed a validated CARMc19 scores with good performance characteristics for predicting the risk of in-hospital mortality. Since the CARMc19 scores place no additional data collection burden on clinicians, it may now be carefully introduced and evaluated in hospitals with sufficient informatics infrastructure.
Article activity feed
-
-
SciScore for 10.1101/2020.11.30.20241273: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are limitations in relation to our study. We identified COVID-19 based on ICD-10 code ‘U071’ which was determined by COVID-19 swab test results (hospital or community) and clinical judgment and so our findings are …
SciScore for 10.1101/2020.11.30.20241273: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization not detected. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are limitations in relation to our study. We identified COVID-19 based on ICD-10 code ‘U071’ which was determined by COVID-19 swab test results (hospital or community) and clinical judgment and so our findings are constrained by the accuracy of these methods [22,23]. This does, however, allow the algorithm to take account of the entry of diagnostic information by the clinician including radiology findings as input variables if the swab result is negative. We used the index NEWS2 data in our models, but vital signs and blood test results are repeatedly updated for each patient according to hospital protocols. Although we developed models using one hospital’s data and validated into another hospital’s data, the extent to which changes in vital signs over time reflect changes in mortality risk need to be incorporated in our models requires further study. Our two hospitals are part of the same NHS Trust and this may undermine the generalisability of our findings, which merit further external validation. Although we focused on in-hospital mortality (because we aimed to aid clinical decision making in the hospital), the impact of this selection bias needs to be assessed by capturing out-of-hospital mortality by linking death certification data and hospital data. CARMc19, like other risk scores, can only be an aid to the decision-making process of clinical teams [11,24] and its usefulness in clinical practice remains to be seen. The next phase of this work is to field test CAR...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
