New Evidence and Design Considerations for Repeated Measure Experiments in Survey Research

We re-examine recent influential claims that repeated measure experimental designs do not introduce bias and offer large precision gains in survey research (Clifford, Sheagley, and Piston 2021). We test these claims by experimentally varying the design of six classic political science experiments across three distinct large samples of U.S. adults (total N = 13, 163). In contrast to the original study, we observe consistent attenuation of treatment effects in repeated measure designs. However, this average design effect is small enough, and the precision gains large enough, that we largely affirm the recommendation to employ repeated measure designs in many practical research applications. We additionally extend the literature on repeated measure designs by exploring how several design considerations affect the bias-precision trade-off, such as the use of within-subject versus between-groups designs, the relative separation of repeated measures within single surveys, and differences in respondent characteristics across sample types.