Mode effects in cognitive assessment: Analysis of an experiment using in-person, video and web data collection

With the rise of web-based data collection, researchers are increasingly exploring how to measure cognition online, including in web surveys and video interviews. This paper provides empirical evidence on mode effects (web, in-person, video interviewing) in the measurement of cognitive abilities using the TestMyBrain Backward Digit Span test (an immediate serial recall task) among adults age 20-40 years in England. Participants completed surveys including the same cognitive test at two time points two weeks apart and were randomly allocated to mode at each wave. The 1,692 wave 1 respondents and 1,510 wave 2 respondents were analysed using mixed effects models to estimate mode effects relating to several assessment outcomes. We found that participation by web was associated with a higher likelihood of an incomplete assessment relative to both in-person and video interviewing, suggesting that the presence of an interviewer – even if not physically present as in video interviewing – enhances the likelihood of a successfully completed test. For participants who completed the cognitive assessment, we found no evidence that the observed mean scores differed by mode, though this appeared to mask some heterogeneity. Although based on small numbers, there was a trend for both web and video respondents to be more likely to obtain the highest test scores than those participating in-person. This is consistent with prior suggestions that individuals in modes where there is limited or no interviewer presence are more likely to cheat. Mean test scores were consistently higher at wave 2 than at wave 1, with this ‘retest gain’ differing little by mode. These findings contribute to the limited literature on mode effects in cognitive assessment and offer valuable insights for researchers designing and analysing mixed-mode studies.