Empirical Review of LLM-driven Classification of Multidimensional Sleep Health Mentions from Free-Text Clinical Notes

Accurate multidimensional sleep health (MSH) information is often fragmented and inconsistently represented within hospital infrastructures, leaving crucial details buried in unstructured clinical notes rather than discrete fields. This inconsistency complicates large-scale phenotyping, secondary analyses, and clinical decision support regarding sleep-related outcomes. In this work, we systematically explore contemporary natural language processing techniques, prompt-based large language models (LLMs) and fine-tuned discriminative classifiers, to bridge this critical gap. We evaluate performance on extracting nine key MSH dimensions (timing, duration, efficiency, sleep disorders, daytime sleepiness, interventions, medication, behavior, and satisfaction) from clinical narratives using public datasets (MIMIC-III derivatives) and an internally annotated pediatric sleep corpus.

Initially, we assess generative LLM performance using dynamic few-shot prompting, analyzing impacts from varying prompt structures, example quantity, and domain-specificity without explicit task-specific fine-tuning. Subsequently, we fine-tune generative LLM architectures on both in-task and out-of-task data to quantify performance improvements and limitations. Lastly, we benchmark these generative approaches against encoder-based discriminative classifiers (ModernBERT), designed to directly estimate binary presence of each MSH class within full clinical notes.

Our experiments demonstrate that fine-tuned discriminative models consistently provide higher classification accuracy, lower inference latency, and more robust span-level identification than either prompted or fine-tuned generative LLMs, given adequate training data. Nonetheless, generative LLMs retain moderate utility in low-data scenarios. Importantly, our results highlight persistent challenges, including difficulty extracting subtle sleep constructs such as sleep efficiency and daytime sleepiness, and biases associated with patient demographics and clinical departments. We conclude by suggesting future research directions: refining span extraction methods, mitigating biases in model performance, and exploring advanced chain-of-thought prompting techniques to achieve reliable, scalable MSH phenotyping within real-world clinical systems.