COVID-19 risk score as a public health tool to guide targeted testing: A demonstration study in Qatar
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (PREreview)
- Evaluated articles (ScreenIT)
Abstract
We developed a Coronavirus Disease 2019 (COVID-19) risk score to guide targeted RT-PCR testing in Qatar. The Qatar national COVID-19 testing database, encompassing a total of 2,688,232 RT-PCR tests conducted between February 5, 2020-January 27, 2021, was analyzed. Logistic regression analyses were implemented to derive the COVID-19 risk score, as a tool to identify those at highest risk of having the infection. Score cut-off was determined using the ROC curve based on maximum sum of sensitivity and specificity. The score’s performance diagnostics were assessed. Logistic regression analysis identified age, sex, and nationality as significant predictors of infection and were included in the risk score. The ROC curve was generated and the area under the curve was estimated at 0.63 (95% CI: 0.63–0.63). The score had a sensitivity of 59.4% (95% CI: 59.1%-59.7%), specificity of 61.1% (95% CI: 61.1%-61.2%), a positive predictive value of 10.9% (95% CI: 10.8%-10.9%), and a negative predictive value of 94.9% (94.9%-95.0%). The concept and utility of a COVID-19 risk score were demonstrated in Qatar. Such a public health tool can have considerable utility in optimizing testing and suppressing infection transmission, while maximizing efficiency and use of available resources.
Article activity feed
-
-
This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/6812983.
In this study, the authors report the development and validation of a prediction model of "risk of COVID-19" in Qatar, with further updating and evaluation of the model. It is not clear what the authors intended to predict, since they only mention "risk of COVID-19", but they later mention that the model was meant to predict "risk of exposure to SARS-CoV-2", which doesn't necessarily imply that the individual will have a positive test since having a positive test is dependent on the course of disease and time of testing is very important for this. Adding confusion to this, the authors report methods consistent with a diagnostic test prediction model development and validation, which suggest …
This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/6812983.
In this study, the authors report the development and validation of a prediction model of "risk of COVID-19" in Qatar, with further updating and evaluation of the model. It is not clear what the authors intended to predict, since they only mention "risk of COVID-19", but they later mention that the model was meant to predict "risk of exposure to SARS-CoV-2", which doesn't necessarily imply that the individual will have a positive test since having a positive test is dependent on the course of disease and time of testing is very important for this. Adding confusion to this, the authors report methods consistent with a diagnostic test prediction model development and validation, which suggest that they only wanted to predict the risk of having a positive SARS-CoV-2 test. Furthermore, the authors mention throughout their title and rest of the manuscript that their model is meant to guide targeted testing in Qatar. In summary, it is not very clear if the authors wanted to 1. develop a prediction model for having a positive SARS-CoV-2 test (which does not necessarily imply having symptomatic disease and would not capture all individuals exposed to SARS-CoV-2), 2. a prediction model for the clinical diagnosis of COVID-19 as the name implies (this would have required at least considering symptoms), or 3. a prediction model of the risk of having been exposed to SARS-CoV-2.
Even when the study has strengths like the large sample size and robust statistical analyses, there are important and serious limitations for this study which could even have important implications that would favor segregation and non-equality of people in Qatar:
1) The authors seem to have used a very limited dataset which only allowed them to include a small set of predictors. Evaluating symptoms would have been a minimal requirement towards developing a tool to perform targeted testing since the WHO has used symptoms to define suspected COVID-19 cases from the start of the pandemic.
2) The predictors identified have no pathophysiological basis (or at least the authors have not adequately described this) for what they are intending to predict. It is very likely that these associations are instead spurious and only reflect confounding due to the characteristics of people who tend to seek diagnostic testing more often. For instance, sex has been included in the model, does this mean that women should not be prioritized for testing? As the authors commented on, their population is predominantly composed of men. The implications of this cannot be underestimated, since prioritizing men for testing would favor sex inequity and could limit the access of women to healthcare. The nationality variable could also only be reflecting the diversity of inhabitants in Qatar, reason why guiding testing according to nationality would also be unethical and could favor discrimination of foreigners in Qatar. It would be very irresponsible in my opinion to use this model to guide diagnostic testing in its current form and without having tested all other important predictors.
Please take these comments very seriously and re-consider if you still want to pursue publication of this paper. Otherwise, extensive modifications are needed since it could be very risky to publish it as it is and there would be a very high risk of the study being retracted due to these important concerns despite the journal where it ends up being published. I am quite sure that the authors had not thought of the implications that their findings could have, and they should not be blamed for it.
The following comments could be useful to the authors to improve their manuscript:
1) The authors do not mention their study design (cross-sectional, cohort, experimental, etc.) at any moment in their methods. Labeling the study design is a current requirement by the Equator-Network recommendations. Note that diagnostic and prognostic prediction models can be derived from 1) cross-sectional 2) cohort, or 3) clinical trials, which is why it is important to fully report this study according to the corresponding study design.
2) The authors may need to review the STROBE statement (https://pubmed.ncbi.nlm.nih.gov/17941715/) and RECORD extension (https://pubmed.ncbi.nlm.nih.gov/26440803/) to report their study since it has characteristics of an observational study using routinely collected health data. Please provide the STROBE checklist + RECORD checklist as supplementary material for peer-review only: https://docs.google.com/viewer?url=http://www.record-statement.org/Files/checklist/RECORD%20Checklist.pdf
3) Additionally, since this study involved the development and validation of a prediction model, the authors would need to report their study according to TRIPOD recommendations (https://pubmed.ncbi.nlm.nih.gov/25560730/). Please provide the TRIPOD checklist as supplementary material for peer-review only to make sure that all elements for adequate reporting have been included.
4) Since the authors are claiming that their study is the first COVID-19 risk score, they need to perform a systematic search of the literature to be able to confidently conclude that. It is not acceptable to say "we believe that" without having made any serious attempts to address that uncertainty. This systematic search of the literature needs to be reported according to as many as possible of PRISMA-S recommendations (https://www.equator-network.org/reporting-guidelines/prisma-s/) even when this is not part of a systematic review per se.
5) The authors mention a prospective validation strategy "The second objective was to assess the prospective performance of this risk sore on epidemic data collected after its derivation", however, their methods are not compatible with having prospectively evaluated the performance of the model since they later declare that they split the sample into 50% for the development of the model and 50% for its validation, which clearly suggests that these analyses were retrospective. Please correct since this is misleading.
5) In case that it was indeed prospective at any point, was this study registered in a publicly available website or was the research protocol made available for public scrutiny? This would have been important to foster transparency, especially since the authors describe a prospective validation component. Please read the following to see what I mean: https://doi.org/10.1371/journal.pmed.1001711
6) Please rename your score, since COVID-19 risk score is very non-specific. At least add "Qatar" to the name of the score in the title as you have done in some sections of the manuscript.
7) Page 5: Please be more specific on the strategy for sampling.
-
SciScore for 10.1101/2021.03.06.21252601: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization This half of the sample was chosen randomly. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your data.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:This study has some limitations. The COVID-19 risk score was derived using the national testing database rather than a nationally representative, probability-based survey of the total population of Qatar. Infection levels and patterns among tested individuals may not necessarily reflect actual levels and …
SciScore for 10.1101/2021.03.06.21252601: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement not detected. Randomization This half of the sample was chosen randomly. Blinding not detected. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
No key resources detected.
Results from OddPub: Thank you for sharing your data.
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:This study has some limitations. The COVID-19 risk score was derived using the national testing database rather than a nationally representative, probability-based survey of the total population of Qatar. Infection levels and patterns among tested individuals may not necessarily reflect actual levels and patterns in the wider population. The score used a small number of demographic variables, but its predictive power might have been enhanced if other variables had been available, such as more socio-demographic indicators. Despite these limitations, the study had important strengths. The testing database encompassed all RT-PCR testing done in Qatar up the present and was massive, including results of over two million tests, representing a majority of the population of Qatar [8, 39]. While other variables in the score might have improved its predictive power, they might have reduced its accessibility and utility for broad use as a tool of public health. In conclusion, the concept and utility of a COVID-19 risk score was demonstrated in a single country. Such public health tool, based on a set of non-invasive and easily captured variables, can help to optimize testing and suppression of infection transmission, while maximizing efficient use of available resources.
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
