Utilizing Qualitative Methods to Detect Validity Issues in Clinical Experience Sampling Methodology (ESM)

Experience sampling methods are used increasingly in clinical research and practice, promising unique insights into people’s daily lives and more accurate, ecologically valid clinical assessment. However, there are rising concerns about the validity of ESM studies due to various measurement challenges, including differences and changes in participants’ concept and scale interpretation (e.g., whether a 4 on a 7-point scale means the same for two individuals), and their interpretation of the study as a whole. Currently, the ESM literature mainly focuses on quantitative solutions. In this paper, we highlight the contribution of qualitative methods to not only detecting the occurrence but also the content of validity issues. We describe how to implement validity checks for ESM studies using focus groups, open-ended items, as well as cognitive and semi-structured interviews. Although these methods are already used in other fields, we present a translation to ESM research and describe implementations suitable for different research stages, from ESM material development to study follow-up. To illustrate the usefulness of these qualitative validity checks, we provide concrete examples from the clinical ESM literature and our own mixed-methods studies. Thereby we hope to encourage clinical researchers and practitioners interested in the implementation of ESM to reflect on how validity issues may impact the conclusions drawn from their collected data.Keywords: intensive longitudinal data, ecological momentary assessment (EMA), qualitative methods, validity checks