A preliminary study of commercially available general-purpose chest radiography artificial intelligence-based software for detecting airspace opacity lesions in COVID-19 patients

  1. Munemura Suzuki
  2. Aruta Niimura
  3. Yusuke Nakamura
  4. Yujiro Otsuka

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Purpose

To validate commercially available general-purpose artificial intelligence (AI)-based software for detecting airspace opacity in chest radiographs (CXRs) of COVID-19 patients.

Materials and Methods

We used the ieee8023-covid-chestxray-dataset to validate commercial AI software capable of detecting “Nodule/Mass” and “Airspace opacity” as regions of interest with probability scores. From this dataset, we excluded computed tomography images and CXR images taken using an anteroposterior spine view and analyzed CXR images tagged with “Pneumonia/Viral/COVID-19” and “no findings.” A radiologist then reviewed the images and rated them on a 3-point opacity score for the presence of airspace opacity. The maximum probability score of airspace opacity for each image was calculated using this software. The difference in each maximum probability for each opacity score was evaluated using Wilcoxon’s rank sum test. The threshold of the probability score was determined by receiver operator characteristic curve analysis for the presence or absence of COVID-19, and the true positive rate (TPR) and false positive rate (FPR) were determined for the individual and overall opacity scores.

Results

Images from 342 patients with COVID-19 and 15 normal images were included. Opacity scores of 1, 2, and 3 were observed in 44, 70, and 243 images, respectively, of which 33 (75%), 66 (94.2%), and 243 (100%), respectively, were from COVID-19 patients. The overall TPR and FPR were 0.82 and 0.13, respectively, at an area under the curve of 0.88 and a threshold of 0.06, while the FPR for opacity score 1 was 0.18 and the TPR for score 3 was 0.97.

Conclusion

Using a public database containing CXR images of COVID-19 patients, commercial AI software was shown to be able to detect airspace opacity in severe pneumonia.

Summary

Commercially available AI software was capable of detecting airspace opacity in CXR images of COVID-19 patients in a public database.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.12.22.21268176: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    EthicsIRB: In accordance with the ethical guidelines for medical and health sciences research involving human subjects10,11, no review by an institutional review board was required.
    Sex as a biological variablenot detected.
    Randomizationnot detected.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Statistical analyses were performed using Python 3.6, Numpy (version 1.18.1), Pandas (version 1.0.2), scikit-learn (version 0.22.2. post 1), and Scipy (1.4.1).
    Python
    suggested: (IPython, RRID:SCR_001658)
    Numpy
    suggested: (NumPy, RRID:SCR_008633)
    scikit-learn
    suggested: (scikit-learn, RRID:SCR_002577)
    Scipy
    suggested: (SciPy, RRID:SCR_008058)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    This preliminary study had several limitations. First, we used public data. The image quality varied, and there were some missing clinical data. In fact, the team that created this database advises not to use them to determine diagnostic performance15. However, because we used public data not used for training for validation, the impact of database bias4 is expected to be small. Second, images were evaluated by a single person. Although there is interobserver variability in the diagnosis of pneumonia16, in this study, each group of confidence scores by a single radiologist could be separated by AI-based probability scores with significant differences. The clinical significance of the overlap is unclear, but it is a potential consideration for future research. Third, we did not evaluate all the lesions extracted by our model. This is because there is no gold standard such as CT, and it is difficult to perform a detailed evaluation because of differences in the image quality and size compared to those of DICOM format images for diagnosis. This is an issue for consideration in future research. In conclusion, our hypothesis that commercially available AI-based software could detect opacity regions in CXRs of COVID-19 patients was proved. We hope to conduct clinical research and develop a prognostic model using the output results of this AI to help in the treatment of COVID-19 infections.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.12.22.21268176v1 on medRxiv