Extracting Symptoms of Psychotic Disorders from Clinical Notes using Natural Language Processing.

Large Language Models (LLMs) are proposed as tools for high-throughput, deep phenotyping of psychiatric disorders. Applied to electronic health records, LLMs could in principle extract patient symptoms, outcome trajectories, risk factors, and treatment history at scale and these, when combined with increasingly available biological data, such as genomics or neuroimaging, could provide powerful resources for health research. Although proof-of-principle LLM-based symptom extractions have been carried out for some medical conditions, the heterogeneous nature of psychiatric disorders, including schizophrenia, requires extensive domain-specific evaluations. Here, we evaluate 14 general-purpose LLMs for extracting eight symptoms of schizophrenia from clinical summaries in a severe mental illness cohort (N = 704). Performance across symptoms was poor to moderate (macro F1 = 0.500 - 0.647), with positive symptoms more accurately extracted than negative symptoms. Few-shot prompting, a common strategy for improving LLM task performance, did not significantly improve these results. Nevertheless, LLM-predicted and gold-standard positive symptoms demonstrated comparable associations with clinical variables in regression analyses. Individual-level extraction errors attenuated group-level associations but did so unevenly across symptom domains. This indicates that general-purpose LLMs may not be able to extract psychosis-related phenotypes from clinical summaries with the quality required for clinical use but might nevertheless be useful for exploratory research on large cohorts. However, closing the performance gap between positive and negative symptom extractions seems essential groundwork to prevent a systematic bias in any LLM-derived characterisations of psychotic symptoms.