Explainable Suicide Phenotyping from Initial Psychiatric Evaluation Notes Using Reasoning Large Language Models

Clinical phenotyping is the process of extracting patient’s observable symptoms and traits to better understand their disease condition. Suicide phenotyping focuses more on behavioral and cognitive characteristics, such as suicide ideation, attempt, and self-injury, to identify suicide risks and improve interventions. In this study, we leveraged the latest reasoning models, namely 4o, o1, and o3-mini, to perform note-level multi-label classification and reasoning generation tasks using previously annotated psychiatric evaluation notes from a safety-net psychiatric inpatient hospital in Harris County, Texas. Compared with the previously finetuned GPT-3.5 model, the out-of-box reasoning models prompted with in-context learning achieved comparable and better performance, with the highest accuracy of 0.94 and F1 of 0.90. We implemented novel clinical justification generation from these models on the traditional classification tasks. This finding marked a promising direction for performing clinical phenotyping that is interpretable and actionable using smaller, efficient reasoning models.