Interobserver Agreement of Lung Ultrasound Findings of COVID ‐19
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (ScreenIT)
Abstract
Lung ultrasound (LUS) has received considerable interest in the clinical evaluation of patients with COVID‐19. Previously described LUS manifestations for COVID‐19 include B‐lines, consolidations, and pleural thickening. The interrater reliability (IRR) of these findings for COVID‐19 is unknown.
Methods
This study was conducted between March and June 2020. Nine physicians (hospitalists: n = 4; emergency medicine: n = 5) from 3 medical centers independently evaluated n = 20 LUS scans ( n = 180 independent observations) collected from patients with COVID‐19, diagnosed via RT‐PCR. These studies were randomly selected from an image database consisting of COVID‐19 patients evaluated in the emergency department with portable ultrasound devices. Physicians were blinded to any patient information or previous LUS interpretation. Kappa values ( κ ) were used to calculate IRR.
Results
There was substantial IRR on the following items: normal LUS scan ( κ = 0.79 [95% CI: 0.72–0.87]), presence of B‐lines ( κ = 0.79 [95% CI: 0.72–0.87]), ≥3 B‐lines observed ( κ = 0.72 [95% CI: 0.64–0.79]). Moderate IRR was observed for the presence of any consolidation ( κ = 0.57 [95% CI: 0.50–0.64]), subpleural consolidation ( κ = 0.49 [95% CI: 0.42–0.56]), and presence of effusion ( κ = 0.49 [95% CI: 0.41–0.56]). Fair IRR was observed for pleural thickening ( κ = 0.23 [95% CI: 0.15–0.30]).
Discussion
Many LUS manifestations for COVID‐19 appear to have moderate to substantial IRR across providers from multiple specialties utilizing differing portable devices.
The most reliable LUS findings with COVID‐19 may include the presence/count of B‐lines or determining if a scan is normal. Clinical protocols for LUS with COVID‐19 may require additional observers for the confirmation of less reliable findings such as consolidations.
Article activity feed
-
-
SciScore for 10.1101/2020.08.16.20176156: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement IRB: This study has received institutional review board (IRB) approval at all participating sites. Randomization Study Procedure: In this study, a total of n=20 LUS scans collected between 3/2020 and 6/2020 were randomly selected from our image database consisting of COVID-19 POCUS images. Blinding The physicians were blinded to any patient information or previous interpretation. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
Software and Algorithms Sentences Resources The physicians were then instructed to independently assess the 20 de-identified studies and input their interpretation using a central electronic database … SciScore for 10.1101/2020.08.16.20176156: (What is this?)
Please note, not all rigor criteria are appropriate for all manuscripts.
Table 1: Rigor
Institutional Review Board Statement IRB: This study has received institutional review board (IRB) approval at all participating sites. Randomization Study Procedure: In this study, a total of n=20 LUS scans collected between 3/2020 and 6/2020 were randomly selected from our image database consisting of COVID-19 POCUS images. Blinding The physicians were blinded to any patient information or previous interpretation. Power Analysis not detected. Sex as a biological variable not detected. Table 2: Resources
Software and Algorithms Sentences Resources The physicians were then instructed to independently assess the 20 de-identified studies and input their interpretation using a central electronic database (REDCap). REDCapsuggested: (REDCap, RRID:SCR_003445)Results from OddPub: We did not detect open data. We also did not detect open code. Researchers are encouraged to share open data when possible (see Nature blog).
Results from LimitationRecognizer: We detected the following sentences addressing limitations in the study:There are several limitations to this study. Our study population was confined to patients who presented to the emergency department or were hospitalized, which limits the generalizability of these findings for providers in the outpatient setting. Although we randomly selected LUS studies from our database, we sampled fewer patients (n=13) than the total number of scans analyzed (n=20). The researchers in this study completed a 1-hour calibration session and had a definition sheet when interpreting images. Therefore, the IRR may be lower for certain findings among less-trained practitioners. Although this study utilized several portable devices, there was variability in the image quality of these devices (particularly when visualizing the pleural line), which may have affected the results regarding pleural thickening or subpleural consolidation. Despite these limitations, this study represents one of the first dedicated investigations into the interobserver agreement of LUS findings for COVID-19. In conclusion, many LUS manifestations for COVID-19 appear to have substantial to moderate IRR across providers from multiple specialties utilizing differing portable devices. More reliable findings included the determination of a normal scan, the presence and location of B-lines, and determining if >=3 B-lines were present. Less reliable findings related to the presence or locations of consolidations, pleural thickening, or effusions. Since presence of B-lines may have diagnostic an...
Results from TrialIdentifier: We found the following clinical trial numbers in your paper:
Identifier Status Title NCT04384055 Enrolling by invitation Predicting Outcomes for Covid-19 Using Sonography Results from Barzooka: We did not find any issues relating to the usage of bar graphs.
Results from JetFighter: We did not find any issues relating to colormaps.
Results from rtransparent:- Thank you for including a conflict of interest statement. Authors are encouraged to include this statement when submitting to a journal.
- Thank you for including a funding statement. Authors are encouraged to include this statement when submitting to a journal.
- No protocol registration statement was detected.
-
