AI in the Experimental Loop: Implications for Replicability in Social Sciences

Integrating generative Artificial Intelligence (AI), in particular Large Language Models, into experimental social sciences presents both powerful opportunities and significant methodological challenges. One major issue is the stochastic nature of the output of those models, which complicates replicability—a foundational principle of scientific research. This paper examines the transition from the traditional Experiment-Subjects Dyad (ESD) experimental design to what we refer to as an Experiment-AI-Subjects Triad (EAIST) experimental design, where AI is employed to generate experimental trials. In the EAIST experimental design, AI can provide adaptive stimuli and generative systems, which may undermine experimental control and threaten the reliability of Human-Robot Interaction studies. We review specific examples, such as emotionally expressive chatbots and GAN-generated facial expressions, and identify the two sources of variability they introduce. We then propose a framework to enhance replicability in AI-driven research, drawing on principles from the Open Science movement. Key strategies include modular testing, parameter fixation, structured prompt engineering, and robust experimental design. We provide a checklist to ensure robustness and replicability of the EAIST experiments. Our recommendations aim to maintain the ecological advantages of AI while reinforcing methodological transparency and scientific rigour.