Attention check items do not bother participants (at least when used in moderation)

It is important for researchers to be able to identify sources of low-quality data in their surveys. At the same time, it is important that the methods researchers use to detect these sources of low-quality data do not result in a substantially worse survey-taking experience for their participants. The purpose of the present project was to investigate how two types of attention check items—infrequency/frequency items (e.g., “I don’t have a first name”) and instructed-response items (e.g., “Choose disagree for this item”)—impact the survey-taking experiences of participants. In Study 1 (N = 400), we found that participants responded most favourably to a survey composed entirely of control items, followed by a survey composed entirely of infrequency/frequency items and, finally, a survey composed entirely of instructed-response items. However, in Study 2 (N = 793) and Study 3 (N = 800), we found that these effects only appeared when over half of the survey consisted of infrequency/frequency items and instructed-response items, respectively. These results suggest that attention check items can negatively impact a participant’s survey-taking experience but only when administered in large quantities. They also suggest that infrequency/frequency items tend to be better received than instructed-response items.