Disagreement between AI and Radiologists: Quantifying Direction and Fragility
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Conventional performance metrics do not indicate whether artificial intelligence (AI)-platforms and radiologists are right or wrong on the same examinations, nor whether their non-agreement is more often beneficial or harmful.
Objective
To develop, illustrate, and apply directional metrics quantifying non-agreement between radiologists and AI-platforms against a common reference standard, and to adapt a fragility-based framework for quantifying the statistical robustness of these metrics.
Methods
We defined Positive Non-Agreement (PNA, AI-platform correct, radiologist incorrect), Negative Non-Agreement (NNA, radiologist correct, AI-platform incorrect), and Net Non-Agreement (NENA, their difference). Metrics were illustrated in three hypothetical scenarios of 1,000 examinations each and applied to three previously published AI-radiologist studies on pulmonary embolism and mammography. Bootstrap confidence intervals were calculated, and a reclassification fragility index (FI) quantified the minimum case reclassification needed to remove statistical support for NENA’s direction.
Results
In illustration, NENA was positive in two scenarios and negative in one. In the three published studies, NENA ranged from −4.18% to +10.74% and decreased consistently with increasing radiologist experience (junior residents, +6.27%; consultants, +1.45%), a gradient driven primarily by declining PNA with modest increase in NNA. The reclassification FI was defined for all three illustrative scenarios (range, 0.10%-1.70%) and for six of the seven comparisons from these studies (range, 0.34%-4.31%), but not for the weakest comparison, whose confidence interval for NENA included zero.
Conclusions
PNA, NNA, and NENA provide a directional, case-level complement to conventional performance metrics for AI-radiologist non-agreement. The accompanying reclassification FI offers a statistically grounded, generalizable method for quantifying their robustness.
Summary Sentence
Positive, Negative, and Net Non-Agreement, directional metrics developed and applied to published datasets comparing artificial intelligence with radiologists, show that the net benefit of artificial intelligence assistance depends on reader experience and clinical setting, and that the statistical robustness of these directional conclusions varies by comparison and sample size, which is information that conventional performance metrics alone do not provide.
Key Points
- Positive Non-Agreement (PNA), Negative Non-Agreement (NNA), and Net Non-Agreement (NENA) are directional metrics that identify whether AI-radiologist non-agreement is more likely to be beneficial or harmful, complementing conventional performance metrics to inform AI validation and monitoring decisions.
- The reclassification fragility index (FI), anchored to statistical significance rather than to the point estimate, quantifies how many case reclassifications would remove support for the direction of NENA.
- Applied to three previously published AI-radiologist datasets NENA ranged from −4.18% to +10.74% and fell from +6.27% to +1.45% with increasing radiologist experience, a gradient driven primarily by declining PNA with modest increase in NNA.
