Open Source LLMs and Latent Concepts in Political Analysis

The proliferation of Large Language Models (LLMs) has transformed text analysis in political science, offering scalable tools for tasks such as annotation and classification. This study explores the potential and limitations of open-source LLMs in coding latent concepts—specifically, policy target groups—from unstructured textual data. Using a dataset documenting algorithmic applications in U.S. federal agencies, we employed two open-source LLMs, Llama-3.1 and Qwen2, and compared their performance to human coders. Despite promising results, LLM responses varied significantly depending on model parameters and prompts, with discrepancies from human coders ranging between 21.7% and over 40%. These findings underscore the necessity of human oversight for nuanced text analysis tasks. We further evaluated the efficacy of semantic similarity tools, finding moderate agreement between human and machine annotations (Cohen's kappa: 0.51–0.57). Our results highlight the utility of open-source LLMs for content analysis while emphasizing the need for continued refinement in prompt engineering and human-machine collaboration to enhance coding accuracy for complex, non-standardized tasks.