Completeness and Quality of Neurology Referral Letters Generated by a Large Language Model for Standardized Scenarios
Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Background
Large Language Models (LLMs) offer promising applications in healthcare, including drafting referral letters. However, access to LLMs specifically designed for medical practice remains limited. While ChatGPT is widely available, its ability to generate comprehensive and clinically appropriate neurology referral letters remains uncertain. This study aimed to systematically evaluate the completeness and quality of neurology referral letters generated by ChatGPT for standardized clinical scenarios.
Methods
Five standardized clinical scenarios representing common neurological complaints (e.g., headache, memory problems, stroke/TIA, tremor, radiculopathy) were used. ChatGPT generated 10 referral letters per scenario using the same prompt. Each letter was evaluated using a predefined scoring rubric assessing for completeness (demographics, chief complaint, history of present illness, physical exam findings, management, and consultation questions), and quality (language level, structure, and letter length), reviewed by a physician dual-board-certified in neurology and family medicine.
Results
ChatGPT reliably included key elements, achieving an average completeness score of 87%. The output demonstrated high language appropriateness (average scores of 92% for language level and 90% for structure). Variability was noted in case management (mean score of 2.18, standard deviation of 0.85). Noticeable gaps in content, particularly in history of present illness and physical findings, were identified in 72% (36/50) of the letters. Document length (250–450 words) was acceptable, though word count often exceeded expectations.
Conclusions
ChatGPT demonstrated high efficiency and utility in generating neurology referral letters but required physician oversight to address variability in case management and minor content gaps. Access to tailored LLMs trained for medical documentation could improve outcomes while safeguarding patient privacy.
Trial registration
Not required (no intervention on human participants)
