Side Effects of Experience Sampling Protocols: A Systematic Analysis of How They Affect Data Quality, Data Quantity & Bias in Study Results

In studies using the increasingly popular Experience Sampling Method (ESM), design decisions are often guided by theoretical or practical considerations. Yet limited empirical evidence exists on how these choices impact data quantity (e.g., response probabilities), data quality (e.g., response latency), and potential biases in study outcomes (e.g., characteristics of study variables).In a preregistered, four-week study (N = 395), we experimentally manipulated two key ESM protocol characteristics for sending ESM surveys: timing (fixed versus varying times) and contingency (directly versus indirectly after unlocking the smartphone). We evaluated the ESM protocols resulting from the combination of these two characteristics with regard to different criteria: As hypothesized for contingency, indirect protocols resulted in higher response probabilities (increased data quantity). But they also led to higher response latencies (reduced data quality). Contrary to our expectations, the combined effect of contingency and timing did not significantly influence response probability. We did also not observe other effects of timing or contingency on data quality. In exploratory follow-up analyses, we discovered that timing significantly affected response probability and smartphone usage behaviors, as measured by screen logs; however, these effects were likely attributable to time of day effects. Notably, self-reported states showed no differences based on the chosen ESM protocol, and similar trends were found when correlating primary outcomes with external criteria such as trait affect and well-being. Based on the study’s findings, we discuss the trade-offs that researchers should consider when choosing their ESM protocols to optimize data quantity, data quality, and biases in study outcomes.