Controlling for careless responding requires causal justification

Careless responding is defined as a pattern of responding to survey items that reflects a latent construct distinct from—and disruptive of—the measurement of the primary construct of interest. In the social science literature, these response patterns are sometimes labeled “insufficient effort,” “unserious,” or “bogus, among other terms.”The issue of such responses has been gaining in relevance as research data are increasingly collected through anonymous online surveys. Standard practice calls for identifying careless responders and excluding them from the data. The common reasoning behind this approach is that such responses can bias “descriptive” statistics (e.g., sample means) and effect size estimates / measures of associations (e.g., correlations and standardised mean difference) as well as increase measurement error. When they confound our causal effect of interest, deleting or adjusting for careless responses is justified. However, using directed acyclic graphs (DAGs), we show that different careless responding patterns can plausibly take on the role of a mediator, a collider, or both simultaneously, among others. Given this, we argue that, in contrast to previous calls, there cannot be a general rule about how to deal with careless responses and that greater attention to their data-generating mechanism from a causal perspective is necessary.