Automated Interpretation of EEG Reports Using a Large Language Model with Structured Confidence Outputs
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To evaluate a large language model (LLM) pipeline for extracting structured diagnostic labels and confidence levels from unstructured free-text EEG reports, addressing the barrier of narrative data analysis.
Methods
We developed a hierarchical schema classifying reports for four abnormality types using a four-point confidence scale. Two certified EEG technologists established ground truth on a diverse dataset of neurologist-authored reports. We implemented a grammar-constrained Mistral-7B pipeline, prompt-tuned to mirror expert annotations, and evaluated it against human benchmarks and classical NLP baselines using core agreement and certainty-adjusted agreement.
Results
Mistral-7B achieved 96% accuracy for overall abnormality detection, approaching the human benchmark (98%) and significantly outperforming baselines. The model successfully identified rare epileptiform abnormalities where traditional models failed and generalized robustly across distinct reporting styles. However, a performance gap persisted in certainty-adjusted agreement, indicating that modeling nuanced confidence remains challenging.
Conclusions
Grammar-constrained LLMs can automate the extraction of structured diagnostic information with near-human accuracy. This pipeline offers a promising tool for standardizing clinical data at scale.
Significance
This study demonstrates a privacy-preserving method to unlock vast archives of clinical EEG reports for research and quality assurance, retaining the critical nuance of diagnostic uncertainty often lost in automated analysis.
HIGHLIGHTS
-
We propose a structured EEG-report classification schema that captures neurologists’ expressed diagnostic confidence.
-
Privacy-oriented local LLMs extract structured labels from routine EEG narratives and are bench-marked vs EEG experts.
-
Automated large-scale classification reaches near-expert accuracy, enabling scalable research datasets and quality assurance.
