Valid Replications Require Valid Methods: Suggestions for Best Methodological Practices with Lab Experiments

This article considers an important but relatively neglected contributor to failures to replicate results from experiments in motivation and emotion – messy methods. By methods, we mean the procedures used to collect data and test hypotheses, which concern issues such as experimental design and validity. We offer a set of guidelines for establishing strong and valid experimental methods for both original research and replications. We first consider the lab room setup, starting with physical environment participants see on the way to the lab as well as the physical environment of the lab room itself. Then, we explain the importance of a cover story and the psychological state of the participants prior to the beginning of the experiment. In addition, we consider experimenters’ and confederates’ behavior and appearance, the need for experimenters to be blind to condition, and the difficulties of having multiple experimenters conduct one experiment. Next, we discuss the construction of strong independent variables, interactions between independent variables, manipulation checks, how the psychological meaning of an independent variable can change over time and place, demand characteristics, and confounds. When considering dependent variables, we explain how to construct sensitive ones, and the importance of pretesting. Then, we apply these guidelines to replications, and finish by considering some data management and statistical issues. We hope this article will be a useful resource for both students and experienced scientists.