Artificial Intelligence-assisted reader evaluation in acute CT head interpretation (AI-REACT): a multireader multicase study
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Non-contrast CT head scans (NCCTH) are the most frequently requested cross-sectional imaging in the Emergency Department. While AI tools have been developed to detect NCCTH abnormalities, most validation studies compare AI to radiologists, with limited evidence on the impact of AI assistance for other healthcare professionals.
Objective
To evaluate whether an AI-powered tool improves the accuracy, speed, and confidence of general radiologists, emergency clinicians, and radiographers in detecting critical abnormalities on NCCTH, and to assess the tool’s stand-alone performance and factors influencing diagnostic accuracy and efficiency.
Methods
A retrospective dataset of 150 NCCTH (52 normal, 98 with critical abnormalities: intracranial haemorrhage, hypodensity, midline shift, mass effect, or skull fracture) was reviewed by 30 readers (10 radiologists, 15 emergency clinicians, 5 radiographers) from four NHS trusts. Each reader interpreted scans first unaided, then with the qER EU 2.0 AI tool, separated by a 2-week washout. Ground truth was established by consensus of two neuroradiologists. We assessed the stand-alone performance of qER and its effect on reader diagnostic accuracy, confidence, and interpretation speed.
Results
The qER algorithm demonstrated strong diagnostic performance across most pathology subgroups (AUC 0.821–0.976). With AI assistance, pooled reader sensitivity for critically abnormal scans increased from 82.8% to 89.7% (+6.9%, 95% CI +1.4% to +10.6%, p<0.001), and for intracranial haemorrhage from 84.6% to 91.6% (+7.0%, 95% CI +3.2% to +10.8%, p<0.001), but specificity decreased from 84.5% to 78.9% (–5.5%, 95% CI –11.0% to –0.09%, p=0.046). Reader confidence AUC did not change significantly. ED clinicians with AI achieved sensitivity comparable to unaided radiologists, with no significant change in specificity.
Conclusion
AI-assisted interpretation increased reader sensitivity for critical abnormalities but reduced specificity. Notably, AI assistance enabled ED clinicians to reach diagnostic sensitivity similar to unaided radiologists, supporting the potential for AI to extend the diagnostic capabilities of non-radiologists. Further prospective studies are warranted to confirm these findings in real-world settings.
Funding
This study was funded by Qure.ai via an NHSX Award
Ethics
The study has been approved by the UK Healthcare Research Authority (IRAS 310995, approved 13/12/2022). The use of anonymised retrospective NCCTH has been authorised by Oxford University Hospitals.
Trial registration number
NCT06018545 .
Research in context
What is already known on this topic
AI-derived algorithms for the detection of pathological findings on non-contrast CT head (NCCTH) images have previously demonstrated strong diagnostic performance when used on retrospective datasets. AI-assisted image interpretation using these algorithms has been shown to enhance the diagnostic performance of general and neuro-radiologists in silico . The potential for AI to enhance the performance of less skilled readers who may encounter and be required to act on these images in clinical practice (e.g. non-specialist radiologists, emergency medicine clinicians and radiographers) is as yet untested, however.
What this study adds
This large multicase multireader study demonstrates that AI-assisted image interpretation may be used to enhance the in silico diagnostic performance of Emergency Department physicians to a level comparable to that of general radiologists.
How this study might affect research, practice or policy
This study raises the possibility that AI-assisted image interpretation could be used to assist non-radiologist clinicians in the safe interpretation of NCCTH scans. Further prospective research is required to test this hypothesis in clinical practice and explore the potential for AI-assisted interpretation to support safe discharge of patients with normal or low-risk scans.
