Characterizing response quality in surveys with multi-item scales: A unified framework

Self-administered surveys are widely used in research across diverse disciplines. One of the major concerns for researchers working with self-administered surveys is the quality of the survey responses. Low-quality responses threaten the validity and replicability of studies and can lead to spurious within-group variability, lower reliability, and potential errors during hypothesis testing. To characterize response quality in a given dataset and detect potentially problematic responses or respondents, the recent literature has proposed many response quality indicators. However, this literature is scattered and suffers from a lack of conceptual and terminological clarity, which limits the practical utility of the indicators that have been proposed. This paper aims to contribute to overcoming these problems. Based on a structured review of existing response quality indicators, we propose a unifying conceptual framework for response quality issues in multi-item surveys. This framework proposes a tripartite distinction between (1) response styles, (2) insufficient effort responding, and (3) invalid responding. It also delineates the relationship between different response quality indicators and the response quality issue(s) they measure. The framework thus establishes a common theoretical point of reference and unified terminology for future work on response quality.