Leveraging Large Language Models on Automating Outpatients’ Message Classifications of Electronic Medical Records

Due to the proliferation of digital equipment and testing, hospitals and clinics generate vast amounts of unstructured textual data on a daily basis. These include electronic health records (EHR), clinical and doctor’s notes, provider-patient communications, and administrative messages. Efficient classification of these messages is critical for enhancing workflow automation, clinical decision-making, and operational efficiency. With rapid progress in natural language processing (NLP), large language models (LLMs) now offer powerful solutions for such classification tasks in healthcare. This paper investigates the application of LLMs in classifying real-world hospital messages using a dataset from a Healthcare system in central Illinois. We compare general-purpose and domain-specific LLMs, evaluating both fine-tuned and few-shot approaches. Our results show that GPT-4o, when fine-tuned within a secure hospital environment, significantly outperforms models like BioBERT and ClinicalBERT. We highlight key challenges such as informal message tone, domain-specific terminology, and classification ambiguity. The study presents practical implications, ethical considerations, and deployment insights that inform the integration of LLMs into clinical workflows.