Fine-Tuned Large Language Models for Detecting Social Isolation from Unstructured Clinical Notes
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
This article is not in any list yet, why not save it to one of your lists.Abstract
Objective
To identify instances of social isolation and social support within unstructured clinical notes by leveraging fine-tuned FLAN-T5-Large, BERT, RoBERTa, and Gemma-2-2B models.
Materials and Methods
The study used annotated clinical note spans containing social context cues from 326,847 adults aged ζ 50 years between 2020 and 2023 to fine-tune each model. Performance was evaluated using Accuracy, Precision, Recall, and Macro-F1 score. A structured prompt was used to instruct the model to perform classification task and mitigate overgeneralization. Performance comparisons across the models assessed sensitivity, robustness, and false positive reduction.
Results
FLAN-T5-Large achieved highest performance, with Macro-F1 of 0.92±0.04, demonstrating balanced results across classes: social isolation (F1 = 0.91±0.03), no social isolation (F1 = 0.94±0.05), and social support (F1 = 0.90±0.04). Gemma-2-2B produced comparable results, with Macro-F1 score of 0.89±0.10. BERT and RoBERTa achieved lower Macro-F1 scores of 0.77±0.17 and 0.80±0.21 respectively, with variability across categories.
Discussion
A major contribution of this work is precise identification of multiple concepts related to social connectedness. By integrating annotated examples of both true and false positives, including negations and contextually ambiguous terms, the model better distinguished relevant social context cues from noise. Training on both social isolation and support provided a dual framework for comparative analyses and patient stratification.
Conclusion
Transformer-based NLP models, particularly FLAN-T5-Large, demonstrated potential for identifying social isolation and social support in clinical text. These findings support the use of generative AI techniques to enhance detection of social isolation from EHRs, advancing context-aware healthcare analytics.
