Expert Discrimination of AI-Generated versus Authentic Radiologic Images: A Multimodal, Pre-Registered Visual Turing Test
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Frontier text-to-image models can synthesise radiologic images of high realism, raising the question of whether expert radiologists can serve as a provenance safeguard for the medical image record.
Methods
We conducted a prospective, pre-registered visual Turing test in which 60 invited Korean board-certified radiology faculty and trainees judged authentic (teaching-repository) and AI-generated radiologic images from a locked pool of 241 displayable cells (82 entities; nine subspecialties; six modalities; 60 readers x 60 trials = 3,600 reader–image observations) produced by two contemporary commercial generators. The primary endpoint was the confidence-weighted, reader-averaged multi-reader multi-case area under the curve for AI versus authentic images, conditional on the locked image pool; the key secondary endpoint was the Faculty-minus-Junior difference under a two one-sided tests equivalence framework. The pre-specified statistical analysis plan was registered on the Open Science Framework before data lock.
Findings
All 60 readers completed the test. The pooled confidence-weighted area under the curve was 0.71 (95% CI, 0.69 to 0.74), above the null value of 0.5 but within the pre-specified modest tier (0.60 to 0.75). The Faculty-minus-Junior contrast was 0.04 (95% CI, −0.02 to 0.10), including zero, and the two one-sided tests established equivalence within the +/−0.10 margin. No reader stratum and no pre-specified sensitivity analysis reached the deployable-classifier threshold (area under the curve >= 0.75).
Interpretation
In this single-country cohort, expert radiologists distinguished frontier-generated from authentic radiologic images only modestly, without a meaningful expertise gradient (equivalence within +/−0.10) and with no reader stratum reaching a standalone provenance safeguard. These findings support radiology AI-literacy training and pipeline-level provenance safeguards rather than reliance on reader judgment, and warrant retesting in an independent reader cohort.
Funding
This research was supported by a grant of the Korea Health Technology R&D Project through the Korea Health Industry Development Institute (KHIDI), funded by the Ministry of Health & Welfare, Republic of Korea (grant number: RS-2025-02213531).
