Measuring Executive Functions Online: Interactive Effects of Experimenter Presence, Instruction Feedback, Session Order, and Task Difficulty

Online cognitive research presents numerous advantages in terms of accessibility and flexibility, often facilitating recruitment and testing. Despite the growing use of online cognitive testing, concerns remain regarding how the unsupervised and uncontrolled environment of this context may be impacting task performance. While various mitigating strategies have been proposed to improve data quality in online testing, their effects have not been consistently evaluated for online cognitive experiments and tend to be assessed in isolation and in single-session studies. To address these limitations, the present study investigated the effects of experimenter presence and instruction feedback on task performance, instruction comprehension, and user experience in an online multi-session study. A total of 109 participants completed one of four conditions where experimenter presence and instruction feedback were manipulated. Each participant was tested over two sessions occurring seven days apart. Participants completed a spatial working memory task in one session and a convergent thinking task in the other, counterbalanced across sessions. Results demonstrated similar instruction comprehension and user experiences across conditions, but significant effects of both experimenter presence and instruction feedback on task performance which varied according to the testing session order, the type of task, and the level of difficulty of the task. The current study adds to the growing literature on the relevance of testing parameters in online cognitive testing by demonstrating how characteristics of the experimental design (type of task, number of sessions) moderate the effects that online parameters have on cognitive performance.