Large Language Models for Variable Relationship Identification in Social Psychology: A Comparative Analysis with Human Experts

Empirical research, particularly in social psychology, centers on formulating testable hypotheses about variable relationships. While traditional approaches rely on theoretical and empirical insights, the process of integrating diverse perspectives and modeling complex social interactions is time-consuming, impacting the efficiency of hypothesis development. Thus, this study evaluates the capacity of large language models (LLMs; Qwen 2.5, Llama 3.1, and GPT-4) to identify variable relationships in social psychology, assessing their reasoning capabilities relative to both domain experts and non-domain experts. To this end, we selected 56 meta-analyses of social psychology published in 2024, from which we extracted 247 variable relationships. We tasked LLMs and human experts to infer variable relationships based on variable definitions and compared their inferences with relationships reported in the meta-analyses, while also examining the impact of task difficulty, self-reported confidence levels, and relationship type on model performance. Key findings indicate that while LLMs and domain experts demonstrated similar performance in identifying simple variable relationships (e.g., linear relationships and difference tests), both exhibited limitations when identifying more complex relationships, particularly moderating effects. Moreover, domain expertise significantly enhanced identification accuracy. Although a correlation was observed between model confidence and accuracy, it did not serve as a reliable predictor. Concurrently, increased task complexity consistently diminished the performance of all LLMs. Future research should focus on enhancing LLMs’ capacity to manage complex variable relationships, as well as exploring the efficacy of human-machine collaboration.