On the statistical analysis of studies with attention checks

Attention checks are often used to identify participants who may be responding carelessly, and to exclude such participants from analysis. There has been little statistical guidance on the analysis of such studies, and on when it is indeed valid to simply exclude inattentive participants. To address this, we first formalize attention checks as measures intended to identify participants whose responses are free of measurement error. Measurement error could arise not only due to careless responding, but also if some participants fail to receive the experimental manipulation in its intended form because they did not attend to its contents. In this context, a causal effect of natural interest is the effect of the independent variable under a hypothetical intervention that also prevents inattentiveness, and hence prevents measurement error. Considering this causal effect, we discuss the statistical assumptions under which it is valid to simply exclude inattentive participants. In randomized experiments, this standard analysis may lead to bias if: (1) the dependent variable affects attentiveness; or (2) there are variables that affect both attentiveness and the dependent variable. The latter assumption is stringent and is likely to be violated in many studies. We suggest a straightforward modification to the standard approach, namely controlling for variables that affect both attentiveness and the dependent variable. This covariate-adjusted approach requires considerably less stringent assumptions. As a worked example, we re-analyze a previously published experiment on a documentary intended to reduce consumption of meat and animal products.