Key Concepts in Online Learning and Decision Making for Just-in-Time Adaptive Interventions

Background: Just-in-time adaptive interventions (JITAIs) provide digital health support that is timely and adaptive: intervention delivery is tailored to rapid changes in internal and external biological and behavioral variables (i.e., “tailoring variables”). JITAIs make decisions on how to tailor support using “decision rules.” Decision rules specify which intervention option should be selected (e.g., whether to send a text-based intervention) based on tailoring variables. JITAI decision rules are typically designed via expert opinion – informed by domain knowledge and statistical analyses of past data – and then kept the same throughout a JITAI’s deployment. Purpose: A challenge for JITAIs is that decision rules can become stale or outdated due to differences in the recipients of a JITAI from one deployment to the next or due to broader societal shifts (e.g., post-pandemic alterations in behavior). As a result, JITAIs may suffer from reduced effectiveness in deployment. In this paper, we describe how online learning and decision-making algorithms create “personalizing JITAIs” (pJITAIs) to address this gap, enabling within-deployment optimization of decision rules based on streaming data collected during deployment. This streaming data provides up-to-date insights on how relationships between tailoring variables, intervention options, and outcomes may differ. Methods: We detail key concepts for online learning and decision-making algorithms, unifying two dominant perspectives in JITAI literature: reinforcement learning (RL) and control engineering. Conclusions: Although RL and control engineering approaches have progressed parallelly in the literature, we elucidate the substantial overlap and unify lexicon to streamline the collaborative design of online learning and decision-making algorithms for pJITAIs.