Mode effects on survey item measurement: A systematic review of the experimental evidence

Survey data are increasingly collected using mixed-mode designs. However, themeasurement of survey items may differ across modes, introducing ‘mode effects’, a type ofsystematic measurement error which can bias analyses of mixed-mode data. While thetheoretical mechanisms giving rise to mode effects have been discussed in detail, theempirical evidence on their occurrence and size is fragmented. In addition, while manyexisting statistical approaches for handling mode effects require unrealistic assumptions,other more suitable approaches remain underutilised due to the need for external evidence onthe magnitude of mode effects. To address this, we conducted a systematic review of theexperimental literature on mode effects. We searched multiple bibliographic databases, greyliterature sources, and implemented backwards and forwards citation screening. Studieseligible for inclusion were (quasi-)experimental, sampled from the general population (orage-, sex-, region-specific strata), and reported mode effect estimates on item measurement.We extracted comprehensive information relating to the study design, sampling, mode effectestimates, and reporting. Ninety experimental studies published between 1967 and 2024 metthe inclusion criteria, which included 4,113 mode effect estimates for 3,545 unique variablesin total. Mode effects were generally small, typically below 0.2 SD. However, larger modeeffects were more commonly observed when modes differed by interviewer involvement orby question delivery (visual vs aural), as well as for sensitive items (e.g., sexual behaviour,social life), which aligns with pre-existing theory on the causes of mode effects. Generally,where mode effects occur, they are item-, mode-, and population-specific. Reporting qualityvaried substantially and insufficient details regarding randomisation compliance, non-response, and uncertainty of estimates were common. We collated all mode effect estimatesinto a free online database and provide a set of recommendations to improve the reporting offuture studies.