Zero-shot extraction of seizure outcomes from clinical notes using generative pretrained transformers
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Purpose
Pre-trained encoder transformer models have extracted information from unstructured clinic note text but require manual annotation for supervised fine-tuning. Large, Generative Pre- trained Transformers (GPTs) may streamline this process. In this study, we explore GPTs in zero- and few-shot learning scenarios to analyze clinical health records.
Materials and Methods
We prompt-engineered LLAMA2 13B to optimize performance in extracting seizure freedom from epilepsy clinic notes and compared it against zero-shot and fine-tuned Bio+ClinicalBERT models. Our evaluation encompasses different prompting paradigms, including one-word answers, elaboration-based responses, prompts with date formatting instructions, and prompts with dates in context.
Results
We found promising median accuracy rates in seizure freedom classification for zero-shot GPT models: one-word – 62%, elaboration – 50%, prompts with formatted dates – 62%, and prompts with dates in context – 74%. These outperform the zero-shot Clinical BERT model (25%) but fall short of the fully fine-tuned BERT model (84%). Furthermore, in sparse contexts, such as notes from general neurologists, the best performing GPT model (76%) surpasses the fine-tuned BERT model (67%) in extracting seizure freedom.
Conclusion
This study demonstrates the potential of GPTs in extracting clinically relevant information from unstructured EMR text, offering insights into population trends in seizure management, drug effects, risk factors, and healthcare disparities. Moreover, GPTs exhibit superiority over task-specific models in contexts with the potential to include less precise descriptions of epilepsy and seizures, highlighting their versatility. Additionally, simple prompt engineering techniques enhance model accuracy, presenting a framework for leveraging EMR data with zero clinical annotation.
Competing Interests
The authors declare no financial or non-financial competing interests.
