Trial and Insight: Combining Quantitative Content Analysis and AI for Experimental Stimulus Generation

Experimental communication research has a fundamental bottleneck: Stimuli are known to vary widely in practice, and concerns of statistical power as well as pragmatic resources place strong limits on the complexity of designs. Maximizing external (and internal) validity therefore requires the paradoxical instrument of highly realistic yet theoretically graded stimuli. The complexity involved in the production of media content so far made this goal prohibitively difficult, but Large Language Models (LLMs) promise to change that. Drawing on an existing content analysis codebook focusing on the identification of scandalization in news articles about climate protests, we leverage generative capabilities of LLMs to produce synthetic stimuli satisfying theoretical variations. These stimuli are validated by human and AI coders on perceived authenticity and the effectiveness of producing variations in target variables. Our findings suggest that LLMs are generally effective in generating experimental stimuli and that quality assessments from GPT4o and human raters are largely aligned.