Investigating the Capability of Large Language Models to Identify Causal Relations in Psychiatric Case Studies: A Methodological Proof of Concept for the Analysis of Psychological Case Formulations

Psychological case formulation, crucial for effective mental health treatment, involves generating causal hypotheses about patient problems. However, the narrative nature of these formulations often complicates their objective assessment. This paper presents a methodological proof of concept investigating the feasibility of using a Large Language Model (LLM) to convert psychological formulations into graph-based representations for preliminary quantitative analysis. By analysing 20 fictitious case studies, the study explored how well LLM-generated graphs matched clinician-developed graphs, focusing on the alignment of nodes and edges. Semantic similarity of node labels, assessed using cosine semantic similarity and binary classification metrics, achieved a high average score of 0.86 (SD = 0.12). Receiver operating characteristic (ROC) analysis, based on binary classification of cosine semantic similarity scores ('similar' vs. 'different'), yielded an area under the curve (AUC) of 0.917 (SE = 0.025, 95% CI [0.868, 0.968], p < 0.001). Causal connections, compared using the Jaccard similarity index, showed a mean index of 0.5 (95% CI [0.42, 0.58]) for the LLM vs. clinician comparison, significantly higher than the LLM vs. random sets comparison (0.11, 95% CI [0.06, 0.17], p < 0.001), and surpassing the pre-registered benchmark index of 0.36 in 80% of the cases. These findings suggest preliminary alignment between LLM-generated and clinician-developed graphs, offering proof of concept for a methodological framework combining natural language processing, graph theory, and semantic analysis in case formulation research. Future research should incorporate real patient data, expand to diverse healthcare domains, and evaluate this method’s potential for advancing personalised and precision mental health care.