Comparison of AI-generated radiology impressions: a multi-stakeholder evaluation

A retrospective, blinded evaluation of 200 oncologic computed tomography reports compared original radiologist-authored impressions, impressions generated by a custom domain-specific AI model fine-tuned on institutional data, and impressions generated by a general-purpose large language model. Ten clinicians, including original radiologists ( n  = 4), independent radiologists ( n  = 3), and oncologists ( n  = 3), rated impressions for completeness, correctness, conciseness, clarity, clinical utility, and patient harm. Original and independent radiologists assigned lower preference to generic model impressions (Cohen’s h 1.04–1.22 and 0.66–0.69, p  < 0.001). Original radiologists slightly preferred their own impressions to the custom model ( h  = 0.18, p = 0.0716), while independent radiologists showed no preference ( h  = −0.03, p = 0.78). Oncologists demonstrated no significant preference among impression types ( h  = 0.04–0.12, all p  > 0.20). Custom model impressions achieved near parity with human impressions; original radiologists rated their own impressions slightly more complete ( r  = 0.22, p  = 0.0016). Generic model impressions were longer (75.1 ± 20.4 words), slightly more complete ( r  = 0.18–0.39, p  < 0.001–0.01), but significantly less concise ( r  = 0.85–0.87, p  < 0.001). Patient harm ratings were uniformly low (likelihood 1.01–1.14; extent 1.05–1.21). Inter-rater reliability ranged from −0.09 to 0.67 ( α  = 0.67 conciseness; α  = −0.09–0.03 clinical utility/correctness).