Detecting Errors in Coronary Computed Tomography Angiography Reports: Comparison Among Three Large Language Models and Human Reader

Background Research on the application of large language models (LLMs) in coronary computed tomography angiography (CCTA) is still in its early stages, with a lack of comprehensive comparative evaluations across multiple LLMs. This study aims to assess the performance of LLMs in identifying errors in CCTA reports, comparing their effectiveness with human reader, and evaluating their time and cost efficiency. Methods In this retrospective study, 600 radiology reports were collected from January to December 2024 at a single institution. A total of 373 errors, categorized into five common types (omission, insertion, spelling, confusion, and other), were intentionally introduced into 300 reports. Other 300 error-free reports were served as the reference standard. Three LLMs (DeepSeek-R1, DeepSeek-32B, and GPT-4o) and one senior radiologist, were tasked with identifying errors. The detection performance and reading time were evaluated using Pearson’s chi-square tests and paired-sample t tests. Results DeepSeek-R1 achieved the highest detection rate among LLMs at 80.7% (301/373, 95% CI: 76.4, 84.4), surpassing GPT-4o (64.1%, 239/373, 95% CI: 59.1, 68.8) and DeepSeek-32B (71.3%, 266/373, 95% CI: 66.5, 75.7). specifically, DeepSeek-R1 outperformed or was non-inferior to the radiologist in omission (74.0% vs. 88.3%, P  = 0.069), insertion (100% vs. 96%, P  = 0.24), spelling (92.9% vs. 78.6%, P  = 0.048), and confusion errors (100% vs. 100%, P  > 0.99), despite a lower total error detection rate compared to radiologist (80.7% vs. 88.7%, P  = 0.006). All LLMs processed markedly faster (6.8–72.4s vs 450.6s; all P  < 0.001) and incurred significantly lower costs ($0.004 - $0.066 vs. $0.692 per report; all P  < 0.001) than the radiologist. Conclusion DeepSeek-R1 can serve as an effective supplementary tool for proofreading in CCTA, offering benefits of efficiency and cost reduction.