Just in Time or Just a Guess? Addressing Challenges in Validating Prediction Models Based on Longitudinal Data

A common goal of researchers using longitudinal data is to develop models that predict emotions orbehaviours, often using passively collected data from smartphone sensors or wearable devices. Afrequent use case for such models is the development of just-in-time adaptive interventions (JITAIs).However, real-world effectiveness depends on rigorous evaluation. Previous research highlightschallenges in selecting appropriate evaluation methods. To address these, we review key pitfalls inpredictive modelling and provide recommendations for avoiding them. We focus on a common problem:the mismatch between development, evaluation and application, and use simulations to illustrate threepitfalls. First, while models may perform well overall (AUC = 0.77), they may lack accuracy in predictingwithin-person change (mean AUC = 0.56, SD = 0.11). For JITAIs, this will prevent the model fromidentifying intervention delivery moments and will only discriminate between individuals. Centeringpredictor variables within individuals improves within-person accuracy, but may reduce overallperformance. Second, ensuring adequate variability in the outcome variable is critical. If outcomesremain stable, frequent prediction may offer little practical benefit. Third, selecting appropriate baselinemodels is essential; models that appear effective may underperform compared to simple baselines (e.g.AUC = 0.77 vs. 0.96). To address these pitfalls, we present recommendations for matching validation andevaluation strategies to the intended use case scenario, and provide a tool that can help researchersinvestigate whether their strategy and goal are misaligned. This can help improve the effectiveness ofpredictive models and increase their utility in real-world applications.