AI in the Experimental Loop: Implications for Replicability in Social Robotics and Social Sciences

Integrating generative Artificial Intelligence (AI), in particular Large Language Models, into social robotics research and experimental social sciences presents both powerful opportunities and significant methodological challenges. One major issue is the stochastic nature of the output of those models, which complicates replicability—a foundational principle of scientific research. This paper explores the transition from the traditional \textit{Experiment-Subjects Dyad} (ESD) experimental design to what we call an \textit{Experiment-AI-Subjects Triad} (EAIST) experimental design, where AI is used to generate experimental trials. In the EAIST experimental design, AI can provide adaptive stimuli and generative systems which may undermine experimental control and threaten the reliability of Human-Robot Interaction studies. We review specific examples, such as emotionally expressive chatbots and GAN-generated facial expressions, and identify the two sources of variability they introduce. We then propose a framework to enhance replicability in AI-driven research, drawing on principles from the Open Science movement. Key strategies include modular testing, parameter fixation, structured prompt engineering, and robust experimental design. Our recommendations aim to maintain the ecological advantages of AI while reinforcing methodological transparency and scientific rigor.