Diagnosis of COVID-19 Using CT image Radiomics Features: A Comprehensive Machine Learning Study Involving 26,307 Patients

  1. Isaac Shiri
  2. Yazdan Salimi
  3. Abdollah Saberi
  4. Masoumeh Pakbin
  5. Ghasem Hajianfar
  6. Atlas Haddadi Avval
  7. Amirhossein Sanaat
  8. Azadeh Akhavanallaf
  9. Shayan Mostafaei
  10. Zahra Mansouri
  11. Dariush Askari
  12. Mohammadreza Ghasemian
  13. Ehsan Sharifipour
  14. Saleh Sandoughdaran
  15. Ahmad Sohrabi
  16. Elham Sadati
  17. Somayeh Livani
  18. Pooya Iranpour
  19. Shahriar Kolahi
  20. Bardia Khosravi
  21. Maziar Khateri
  22. Salar Bijari
  23. Mohammad Reza Atashzar
  24. Sajad P. Shayesteh
  25. Mohammad Reza Babaei
  26. Elnaz Jenabi
  27. Mohammad Hasanian
  28. Alireza Shahhamzeh
  29. Seyed Yaser Foroghi Gholami
  30. Abolfazl Mozafari
  31. Hesamaddin Shirzad-Aski
  32. Fatemeh Movaseghi
  33. Rama Bozorgmehr
  34. Neda Goharpey
  35. Hamid Abdollahi
  36. Parham Geramifar
  37. Amir Reza Radmard
  38. Hossein Arabi
  39. Kiara Rezaei-Kalantari
  40. Mehrdad Oveisi
  41. Arman Rahmim
  42. Habib Zaidi

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Purpose

To derive and validate an effective radiomics-based model for differentiation of COVID-19 pneumonia from other lung diseases using a very large cohort of patients.

Methods

We collected 19 private and 5 public datasets, accumulating to 26,307 individual patient images (15,148 COVID-19; 9,657 with other lung diseases e.g. non-COVID-19 pneumonia, lung cancer, pulmonary embolism; 1502 normal cases). Images were automatically segmented using a validated deep learning (DL) model and the results carefully reviewed. Images were first cropped into lung-only region boxes, then resized to 296×216 voxels. Voxel dimensions was resized to 1×1×1mm 3 followed by 64-bin discretization. The 108 extracted features included shape, first-order histogram and texture features. Univariate analysis was first performed using simple logistic regression. The thresholds were fixed in the training set and then evaluation performed on the test set. False discovery rate (FDR) correction was applied to the p-values. Z-Score normalization was applied to all features. For multivariate analysis, features with high correlation (R 2 >0.99) were eliminated first using Pearson correlation. We tested 96 different machine learning strategies through cross-combining 4 feature selectors or 8 dimensionality reduction techniques with 8 classifiers. We trained and evaluated our models using 3 different datasets: 1) the entire dataset (26,307 patients: 15,148 COVID-19; 11,159 non-COVID-19); 2) excluding normal patients in non-COVID-19, and including only RT-PCR positive COVID-19 cases in the COVID-19 class (20,697 patients including 12,419 COVID-19, and 8,278 non-COVID-19)); 3) including only non-COVID-19 pneumonia patients and a random sample of COVID-19 patients (5,582 patients: 3,000 COVID-19, and 2,582 non-COVID-19) to provide balanced classes. Subsequently, each of these 3 datasets were randomly split into 70% and 30% for training and testing, respectively. All various steps, including feature preprocessing, feature selection, and classification, were performed separately in each dataset. Classification algorithms were optimized during training using grid search algorithms. The best models were chosen by a one-standard-deviation rule in 10-fold cross-validation and then were evaluated on the test sets.

Results

In dataset #1, Relief feature selection and RF classifier combination resulted in the highest performance (Area under the receiver operating characteristic curve (AUC) = 0.99, sensitivity = 0.98, specificity = 0.94, accuracy = 0.96, positive predictive value (PPV) = 0.96, and negative predicted value (NPV) = 0.96). In dataset #2, Recursive Feature Elimination (RFE) feature selection and Random Forest (RF) classifier combination resulted in the highest performance (AUC = 0.99, sensitivity = 0.98, specificity = 0.95, accuracy = 0.97, PPV = 0.96, and NPV = 0.98). In dataset #3, the ANOVA feature selection and RF classifier combination resulted in the highest performance (AUC = 0.98, sensitivity = 0.96, specificity = 0.93, accuracy = 0.94, PPV = 0.93, NPV = 0.96).

Conclusion

Radiomic features extracted from entire lung combined with machine learning algorithms can enable very effective, routine diagnosis of COVID-19 pneumonia from CT images without the use of any other diagnostic test.

Article activity feed

  1. ScreenIT

    SciScore for 10.1101/2021.12.07.21267367: (What is this?)

    Please note, not all rigor criteria are appropriate for all manuscripts.

    Table 1: Rigor

    EthicsIRB: Our study was approved by the local ethics committees, and written informed consent was waived owing to its retrospective nature.
    Consent: Our study was approved by the local ethics committees, and written informed consent was waived owing to its retrospective nature.
    Sex as a biological variablenot detected.
    RandomizationFirst of all, the entire dataset (26,307 patients including 15,148 COVID-19 and 11,159 non-COVID-19) being randomly split into 70% (18,414 patients) and 30% (7,893 patients) for the training and test sets, respectively.
    Blindingnot detected.
    Power Analysisnot detected.

    Table 2: Resources

    Software and Algorithms
    SentencesResources
    Radiomics feature extraction was performed using the Pyradiomcis library 50, which has been standardized according to the Image Biomarker Standardization Initiative (IBSI) 51.
    Image Biomarker Standardization Initiative
    suggested: None
    All multivariate analysis steps were performed in the Python open-source library Scikit-Learn 53.
    Python
    suggested: (IPython, RRID:SCR_001658)

    Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).

    Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:
    Our efforts faced a number of limitations, some of which were addressed and considered. For example, motion artefacts are inescapable when patients are undergoing CT scans. Hence, we excluded these patients as they had an overlapping region of pneumonia in their chest image. Second, not all patients were tested for COVID-19 RT-PCR as some of them were included in our study only by their positive CT signs. Thus, we attempted to overcome this limitation using different model evaluation scenarios (e.g. choosing patients with positive RT-PCR as the testing set) in order to reach a reproducible model for further studies. Third, we did not include clinical or laboratory data. However, these data have been shown to be linked with CT image features 36, 65. Finally, image acquisition parameters were undoubtedly distinct in each center and affected radiomic features. Further studies should be performed using harmonized features between different centers.

    Results from TrialIdentifier: No clinical trial numbers were referenced.

    Results from Barzooka: We did not find any issues relating to the usage of bar graphs.

    Results from JetFighter: We did not find any issues relating to colormaps.

    Results from rtransparent:
    • Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
    • Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
    • No protocol registration statement was detected.

    Results from scite Reference Check: We found no unreliable references.

    About SciScore

    SciScore is an automated tool that is designed to assist expert reviewers by finding and presenting formulaic information scattered throughout a paper in a standard, easy to digest format. SciScore checks for the presence and correctness of RRIDs (research resource identifiers), and for rigor criteria such as sex and investigator blinding. For details on the theoretical underpinning of rigor criteria and the tools shown here, including references cited, please follow this link.

  2. Version published to 10.1101/2021.12.07.21267367v1 on medRxiv