A meta-analysis of the impact and heterogeneity of explicit demand characteristics

Demand characteristics are a fundamental methodological concern in experimental psychology. Yet, little is known about the direction, magnitude, and consistency of their effects. We conducted a three-level meta-analysis of 252 effect sizes from 52 studies that manipulated explicit demand characteristics (EDCs). On average, EDC’s led to small overall increases in hypothesis-consistent responding (g = 0.21, 95% CI [0.12, 0.31]). However, the effects were heterogeneous (between-study τ = 0.28; within-study σ = 0.18), with the prediction interval ranging from g = 0.89 (a large increase in hypothesis-consistent responding) to g = -0.46 (a moderate decrease in hypothesis-consistent responding). Consistent with previous theorizing, the observed and estimated distribution of these effects suggest that demand characteristics can create false positives, false negatives, upward bias, and downward bias. These unpredictable inferential consequences suggest that further research is needed to test mechanisms theorized to underlie the effects of demand characteristics.