Dual reinforcement-learning network modules for modeling decision-making with multiple strategies
Curation statements for this article:-
Curated by eLife
eLife Assessment
This valuable study presents a computational model of meta-learning that bridges two known neural mechanisms of learning: through activation dynamics and through synaptic plasticity. The evidence that the model is a good match to data from a mouse perceptual decision-making task is convincing, but the evidence that it also matches data from a widely studied human multistep decision-making task is incomplete. This work will be of broad interest to neuroscientists and cognitive scientists studying reinforcement learning and decision making.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
Abstract
Animals and humans use multiple behavioral strategies to perform tasks. However, neural implementations of multiple strategies remain elusive, as some studies propose distinct pathways, while others observe overlapping brain regions associated with strategies. We propose a hybrid deep reinforcement learning (H-DRL) method, in which one network model implements model-free and inference-based behaviors through synaptic plasticity and recurrent activity. H-DRL uses a single updating rule and switches the strategy according to task demands without an explicit arbitrator. H-DRL reproduced mixed strategies of humans in a two-step task. In the mouse perceptual decision-making task, H-DRL adapted the recurrent dynamics with rich learning when the task condition required inference-based behavior, while adopting model-free behavior with lazy learning for a simple condition. The activity of H-DRL units showed condition-dependent maintenance of previous events, consistent with orbitofrontal cortical activity in mice. Our model provides a unified view that one cortical network automatically determines strategies in use depending on task conditions.
Article activity feed
-
eLife Assessment
This valuable study presents a computational model of meta-learning that bridges two known neural mechanisms of learning: through activation dynamics and through synaptic plasticity. The evidence that the model is a good match to data from a mouse perceptual decision-making task is convincing, but the evidence that it also matches data from a widely studied human multistep decision-making task is incomplete. This work will be of broad interest to neuroscientists and cognitive scientists studying reinforcement learning and decision making.
-
Reviewer #1 (Public review):
Maeda et al. propose an algorithm for meta reinforcement learning (Meta RL) that they term "hybrid deep reinforcement learning" (HDRL), which combines two distinct learning mechanisms: one that occurs through continual synaptic plasticity (via backpropagation-through-time), and the other that occurs through recurrent dynamics (i.e., in-context learning). The standard approach to Meta-RL, which I will refer to here as "RL2", is one that involves learning only through recurrent dynamics; this model has found recent application to understanding meta reinforcement learning in the brain. One downside of RL2 as a model of meta-learning in the brain is that, in learning only through recurrent dynamics, RL2 essentially throws the baby out with the bathwater by assuming synaptic plasticity does not have a role in …
Reviewer #1 (Public review):
Maeda et al. propose an algorithm for meta reinforcement learning (Meta RL) that they term "hybrid deep reinforcement learning" (HDRL), which combines two distinct learning mechanisms: one that occurs through continual synaptic plasticity (via backpropagation-through-time), and the other that occurs through recurrent dynamics (i.e., in-context learning). The standard approach to Meta-RL, which I will refer to here as "RL2", is one that involves learning only through recurrent dynamics; this model has found recent application to understanding meta reinforcement learning in the brain. One downside of RL2 as a model of meta-learning in the brain is that, in learning only through recurrent dynamics, RL2 essentially throws the baby out with the bathwater by assuming synaptic plasticity does not have a role in moment-by-moment learning. This study therefore makes a valuable contribution in showing how both synaptic plasticity and recurrent dynamics can be combined in a single model. Importantly, this model does not require an explicit mechanism for mediating between these two learning mechanisms; rather, the authors show that the task (and meta-task) itself determines which mechanisms the model comes to rely on. They then show that their HDRL model can better explain differences in behavioral strategies across conditions of a perceptual decision-making task, and use in silico perturbations to establish that these differences in strategies are due to the distinct contributions of their model's two learning mechanisms. Finally, the authors provide evidence that the neural population activity of mice recording during each task has a consistent structure as the activity of artificial neurons in their HDRL model, but not to the standard RL2 model, which always relies on learning through recurrent activity. This study offers an important extension of the standard method for Meta RL, to one that is both more biologically plausible (i.e., because it seems unreasonable for plasticity to only occur periodically as it does in RL2) and more computationally interesting (i.e., as it explains how these two distinct learning strategies may be deployed automatically based on task demands).
The paper's conclusions seem fairly well supported by their analyses; however, there were a few details that need to be clarified.
(1) There are many architectural differences between the H-DRL vs RL2 model, which precludes a true comparison between the two models solely on the addition of Weight RL. The implementation of H-DRL is relatively stripped down compared to RL2, in terms of using Elman rather than LSTM, standard RL vs A2C, only positive activations allowed, and Q-learning instead of policy gradient. Because of these implementation details (none of which seem essential to the 'real' difference between these models), it is difficult to know exactly how these models differ. E.g., if the differences in behavior are in part due to the differences in architecture between HDRL and RL2 rather than to the differences in learning mechanisms.
(2) The conclusion that mouse behavior is more consistent with HDRL than with RL2 seems likely but is not sufficiently supported by the analyses. One thing I wondered was whether all model instances were truly converged; RL2 will of course take much longer than HDRL to learn since it is only updated per session. The authors' claim that mouse behavior is more like HDRL than RL2 seems to be based on the BIC comparison in Figure 2f, but insufficient detail is given to understand what this analysis is. Figure 2e, by comparison, seems to present all three models as similar (though this comparison is difficult to make given the different y-axes).
(3) The interpretation of the two-step task feels incomplete, and potentially in conflict with the interpretation of HDRL in the rest of the paper. When trained on the two-step task, HDRL is presented as reproducing a mixture of model-based and model-free behavior. What does the RL2 model do in this context? Later, on the perceptual decision-making task, HDRL seems to be presented as being model-based-like (and using recurrent activity) in one context and model-free-like (and using plasticity) in the other. Is it the case that in the two-step task, HDRL is simply doing both? As is, it's unclear to me how I should interpret the analyses related to the two-step task.
-
Reviewer #2 (Public review):
Strengths
The manuscript claims that standard methods for deep recurrent RL ("meta-RL"; Wang et al., 2017) provide a poor match to behavior data from humans and animals in many situations, including the two-step task and a mouse behavior dataset previously published by these authors. This negative story is solid and convincing.
The manuscript refines this model by introducing an online variant of deep recurrent RL. This is a good idea for many reasons. One compelling one, which the authors could choose to emphasize more, is that neurobiology does not respect a strict separation between "online" periods during which weights do not change and "offline" periods in which they do. Both weights and activities are changing all the time. Another is that the behavior literature provides abundant reason to believe in …
Reviewer #2 (Public review):
Strengths
The manuscript claims that standard methods for deep recurrent RL ("meta-RL"; Wang et al., 2017) provide a poor match to behavior data from humans and animals in many situations, including the two-step task and a mouse behavior dataset previously published by these authors. This negative story is solid and convincing.
The manuscript refines this model by introducing an online variant of deep recurrent RL. This is a good idea for many reasons. One compelling one, which the authors could choose to emphasize more, is that neurobiology does not respect a strict separation between "online" periods during which weights do not change and "offline" periods in which they do. Both weights and activities are changing all the time. Another is that the behavior literature provides abundant reason to believe in short-timescale innate reward-seeking mechanisms, which offline batch-based RL does not provide.
The manuscript shows that this online deep recurrent RL can provide a good match to their mouse behavior data. This involves a 2AFC perceptual decision-making task with strong temporal dependencies between the stimuli. In one condition, the stimulus on a particular trial is very likely to be identical to the one on the previous trial and only rarely switches; in another, the stimulus is very likely to differ, and only rarely repeats. Mice find the first condition much easier to learn, perhaps because of an innate win-stay/lose-switch bias which is helpful in the first but harmful in the second.
It also provides evidence that online deep recurrent RL can provide a good match to neural data from the OFC in this task. In the "repeat" condition, their model learns the task largely in weights, which is possible because the weight learning rule is hard-coded to be win-stay/lose-switch. In the "alternate" condition, the model must learn it in activities. Broadly consistent with this, mice in the "alternate" condition, but not the "repeat" condition, show strong task-based representations in the OFC.
Taken together, these results are a meaningful advance in applying the "NeuroAI" research program to the case of trial-by-trial reward learning.
Weaknesses
(1) The key innovation of this manuscript is replacing batch-based recurrent deep RL (which they call "meta RL"), in which weights are updated once per batch via backpropagation through time, with a version of online recurrent deep RL in which weights are updated after every trial.
It is difficult for readers to understand how the version of online deep RL being used works, how it compares to other possibilities, and why it was chosen. The manuscript describes it as truncated backprop through time (TBPTT, equation 9). It is not clear from the equation what the truncation window is. If it is only a single trial, then it is unclear how a network trained in this way manages to do the representation learning necessary to do anything model-based at all.
Various other techniques for real-time recurrent learning exist, many under the heading of "biologically plausible" deep learning. These include techniques involving synaptic eligibility traces (e.g., Murray et al., 2019; Bellec et al., 2020) and possibly others.
The manuscript calls this "Hybrid Deep RL". This name is confusing to me and may be confusing to readers. If this method is a "hybrid" between deep RL and some other thing, it's not clear to me what that other thing might be. It seems rather to be a variant of deep RL. I'd suggest a name that's clearer about that and more specific, perhaps "Online" or "Real-time" Deep RL, or similar. The manuscript also describes it in places as a "two-module" system, which is confusing to readers because it has one module: a single RNN with a single learning rule.
The manuscript would benefit greatly from a more careful discussion of these issues and a positioning of the model within the literature. What is lost when giving up BPTT? (long-timescale representation learning, which is often crucial for RL). What is gained? (learning rules that are local in time and space, and therefore more plausible as mechanistic models of biological learning). Where can we expect this method to fail? (tasks with temporal dependencies that are meaningfully longer than the TBPTT window). What alternatives exist? (eligibility trace methods, etc)
(2) The manuscript claims that their model "reproduced mixed strategies of humans in a two-step task". This portion of the paper is brief and unconvincing.
The key diagnostic plot, the trial-history regression in figure 1e, shows behavior that looks almost entirely model-free. There is little, if any, separation between the lines for "common" and "uncommon" within either of "rewarded" or "unrewarded". The bar plot does show an apparent difference, but these one-trial-back analyses are subject to many confounds.
Human studies involving this task typically involve each participant performing only a single session lasting ~100s of trials. When participants are run for multiple sessions, behavioral strategies change, becoming substantially more "model-based" (Economides et al., 2015). If I understand correctly, the model here is trained for 100 sessions of 100 trials, and the data we are shown comes only from the last 25. A fairer comparison would be to show only the first, or first few, of these sessions.
I imagine that H-DRL is unlikely to show much of a model-based pattern within the first session, as it will require averaging over many trials to learn to sculpt activity dynamics appropriately. By the same token, I am skeptical that H-DRL is plausible as a mechanistic account of the human two-step task - instead, I think it is much more likely that explicit working memory mechanisms are involved (e.g., Otto et al., 2013). H-DRL may be a better model of the two-step task as implemented in animals (Miller et al, 2017; Miranda et al., 2020; Akam et al., 2021).
The manuscript claims that human subjects on the two-step task typically use a mixture of both model-based and model-free strategies, and that neurobiology supports a division between the two. Both of these points are controversial, at least: da Silva and Hare (2020) present a compelling argument that humans are primarily model-free, and neural recordings dating back to the very first two-step task paper (Daw et al., 2011) show overlapping signals.
I do think it might be possible to address this with more careful writing and further simulations. But I do not think this is necessary. The rhetorical role of the two-step task in this manuscript seems really just to be illustrating that H-DRL can exhibit different online learning strategies in activities and in weights. This point is made far more convincingly by the analysis of the mouse perceptual decision-making task. I'd recommend simply dropping the two-step task, and its associated claims, from the manuscript.
-
Reviewer #3 (Public review):
Summary:
This paper proposes and tests a modified reinforcement learning model that can express different decision-making strategies which spontaneously emerge when performing a task and match real animal behavior. A recurrent neural network (RNN) is trained in the style of prior work on meta reinforcement learning (e.g. Wang et al. 2016, 2018), where an initial algorithm is used to train the network and a second algorithm emerges later from the recurrent dynamics of the network. These are termed weight-RL and recurrent-RL, respectively. By making several adjustments to the meta-RL training procedure, the authors find that their version (called hybrid deep reinforcement learning, or H-DRL) shows a better match to real animal behavior and orbitofrontal cortical activity dynamics. Overall, this paper is an …
Reviewer #3 (Public review):
Summary:
This paper proposes and tests a modified reinforcement learning model that can express different decision-making strategies which spontaneously emerge when performing a task and match real animal behavior. A recurrent neural network (RNN) is trained in the style of prior work on meta reinforcement learning (e.g. Wang et al. 2016, 2018), where an initial algorithm is used to train the network and a second algorithm emerges later from the recurrent dynamics of the network. These are termed weight-RL and recurrent-RL, respectively. By making several adjustments to the meta-RL training procedure, the authors find that their version (called hybrid deep reinforcement learning, or H-DRL) shows a better match to real animal behavior and orbitofrontal cortical activity dynamics. Overall, this paper is an improvement on the prior meta-RL model, but it is unclear what neuroscientists should take away from this.
Strengths:
(1) The authors introduce a model, H-DRL, whose main strength is that it can better approximate animal behavior and neural activity on specific tasks compared to its predecessor, meta-RL. This could potentially be important for interpreting the algorithms underlying animal learning.
(2) The changes that have been made from meta-RL are not overly complex.
(3) The model is assessed using two types of decision-making tasks that are similar to many of the tasks used in rodent behavioral and systems neuroscience.
(4) In addition to behavioral data, RNN unit activity is compared directly with single unit orbitofrontal cortical activity. This makes the model appealing for neuroscience applications.
Weaknesses:
(1) In the introduction, the authors mention that other extensions of meta-RL have been tried. How does H-DRL compare with these other modifications?
(2) The authors try to convey the impact of their model in the introduction by stating that, unlike meta-RL, "it embodies a fundamentally different theoretical claim in the context of neuroscience". It is not clear what the fundamentally different theoretical claim is. While meta-RL implements weight updates over longer timescales, H-DRL implements weight updates after each trial. Is this the fundamentally different theoretical claim? If so, that does not seem like a novel concept for behavioral and systems neuroscience.
(3) It is not clear what inspired the specific changes to the meta-RL training procedure. Were they decisions based on incremental trial and error to intentionally reproduce the empirical behavioral and neural results, or were they based on prior logic?
(4) Regarding the inconsistency between OFC neurons and H-DRL in previous outcome encoding (Figure 5c), it would be helpful to speculate on why this happens because it seems to be the only place where H-DRL fails.
(5) Figure 5e seems tangential to the overall message of the paper. The paper is about H-DRL as a superior approach to modelling real behavior and neural activity compared to meta-RL. But this panel, which is the last panel in the paper, is not directly relevant because there is no comparison with the model.
-
-
-
-