Development and validation of early warning score systems for COVID‐19 patients
This article has been Reviewed by the following groups
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
- Evaluated articles (ScreenIT)
Abstract
COVID‐19 is a major, urgent, and ongoing threat to global health. Globally more than 24 million have been infected and the disease has claimed more than a million lives as of November 2020. Predicting which patients will need respiratory support is important to guiding individual patient treatment and also to ensuring sufficient resources are available. The ability of six common Early Warning Scores (EWS) to identify respiratory deterioration defined as the need for advanced respiratory support (high‐flow nasal oxygen, continuous positive airways pressure, non‐invasive ventilation, intubation) within a prediction window of 24 h is evaluated. It is shown that these scores perform sub‐optimally at this specific task. Therefore, an alternative EWS based on the Gradient Boosting Trees (GBT) algorithm is developed that is able to predict deterioration within the next 24 h with high AUROC 94% and an accuracy, sensitivity, and specificity of 70%, 96%, 70%, respectively. The GBT model outperformed the best EWS (LDTEWS:NEWS), increasing the AUROC by 14%. Our GBT model makes the prediction based on the current and baseline measures of routinely available vital signs and blood tests.
Article activity feed
-
-
SciScore for 10.1101/2020.11.04.20225904: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:However, limitations of using death as an outcome measure include that the score may be identifying an early sign of an already irreversible process, and therefore early identification of this may offer limited opportunity for clinical intervention. By contrast, our COVID-19 focused outcome measure provides a clinically useful and …
SciScore for 10.1101/2020.11.04.20225904: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
NIH rigor criteria are not applicable to paper type.Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:However, limitations of using death as an outcome measure include that the score may be identifying an early sign of an already irreversible process, and therefore early identification of this may offer limited opportunity for clinical intervention. By contrast, our COVID-19 focused outcome measure provides a clinically useful and actionable warning which may help clinicians and healthcare system managers to preempt shortages and optimise resource allocation in a pandemic context. The difference in performance between our model and the EWS could be partially explained by the fact that these EWS were developed and optimised to detect ward patients’ deterioration against different outcomes. Nonetheless, these EWS represent the current standard of care for COVID-19 patients, and we took action to mitigate these effects by optimising each EWS threshold for our COVID-19 inpatient cohort. A strength of our machine learning approach is its interpretability, using methods employed elsewhere in clinical practice [32] and shown able to attain patient and clinician trust. The three selected models (GBT, random forest and linear regression) permit querying of variables’ weights and presentation in an explainable way. This ability to make sense of the algorithm decision-making process has repeatedly been described as a critical factor in increasing technology uptake in clinical practice [33]. Moreover, our feature sets are oriented around routinely collected clinical data collected within...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
