On the statistical analysis of studies with attention checks

Attention checks are often used to identify and exclude participants who may be responding carelessly. There has been little statistical guidance on the analysis of such studies, and on when it is indeed valid to simply exclude inattentive participants. To address this, we first formalize attention checks as measures intended to identify participants whose responses are free of measurement error. Measurement error could arise not only due to careless responding, but also if some participants fail to receive the experimental manipulation in its intended form because they did not attend to its contents. We discuss the statistical assumptions under which it is valid to simply exclude inattentive participants. In randomized experiments, this standard analysis may lead to bias if: (1) the dependent variable affects attentiveness; or (2) there are variables that affect both attentiveness and the dependent variable. The latter assumption is stringent and is likely to be violated in many studies. We suggest a straightforward modification to the standard approach, namely controlling for variables that affect both attentiveness and the dependent variable. This covariate-adjusted approach requires considerably less stringent assumptions. In two worked examples, we re-analyze previously published experiments on: (1) a documentary intended to reduce consumption of meat and animal products; and (2) flag priming effects on political conservatism.