Performance of a Chest Radiograph AI Diagnostic Tool for COVID-19: A Prospective Observational Study
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Article activity feed
-
-
SciScore for 10.1101/2021.06.04.21258316: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Ethics IRB: This study was approved by the University of Minnesota institutional review board ( STUDY 00011158). Sex as a biological variable From Author: An equity analysis by sex was conducted within this study. Randomization For COVID-19 negative cases , we collected cases and frontal images combined from: ( 1 ) 2011 – 2016 MIMIC-CXR12 ( random sample of 23,611 images); and ( 2 ) Open-I 2013 IU Chest X-Ray Collection13 ( random sample of 3,814 images). Blinding not detected. Power Analysis Power analysis and Statistical Considerations. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open …
SciScore for 10.1101/2021.06.04.21258316: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Ethics IRB: This study was approved by the University of Minnesota institutional review board ( STUDY 00011158). Sex as a biological variable From Author: An equity analysis by sex was conducted within this study. Randomization For COVID-19 negative cases , we collected cases and frontal images combined from: ( 1 ) 2011 – 2016 MIMIC-CXR12 ( random sample of 23,611 images); and ( 2 ) Open-I 2013 IU Chest X-Ray Collection13 ( random sample of 3,814 images). Blinding not detected. Power Analysis Power analysis and Statistical Considerations. Table 2: Resources
No key resources detected.
Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:For example, a recent review of 62 AI models for COVID-19 from biomedical imaging found significant limitations in AI models published to date and nearly all have been designated as having high bias.5 These biases include: the lack of external validation, lack of equity analysis by race and gender, lack of reporting patient demographics, inadequate number of images, lack of reporting of real-time performance, and the utilization of “unrealistic” training datasets which fail to represent the environment where the model will ultimately be deployed. Finally, albeit unlikely, it is possible that false positives by AI are actually patients with COVID-19 that had a negative PCR test despite being COVID-19 positive. The current sensitivity of rapid PCR testing varies significantly based on the viral load of a patient. Patients with viral load cycle threshold (Ct) levels < 25 have a sensitivity of 90%; however, in patients with lower viral loads (higher Ct) PCR sensitivity drops to 76%.31 A question raised by our findings is what is the performance bar for AI models in clinical decision diagnostic support? Does a model with an AUROC of 0.7 or 0.8 not add additional information that the clinician can integrate into decision making? Similar to how an elevated white blood cell count (AUROC 0.70-0.7532,33) in a patient with right lower quadrant tenderness adds diagnostic information towards a work-up for appendicitis. Our findings suggest that AI analysis of chest x-rays alone is not ade...
Results from TrialIdentifier: No clinical trial numbers were referenced.
Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
Results from scite Reference Check: We found no unreliable references.
-
