A neural network model of when to retrieve and encode episodic memories
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
This paper addresses an important problem in control of episodic memory. This paper develops a computationally-based proposal about how semantic, working memory, and episodic memory systems might learn to interact so that stored episodic memories can optimally contribute to reconstruction of semantic memory for event sequences. This is an understudied area and this present work can make a major theoretical contribution to this domain with new predictions. The reviewers were positive about the contribution.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
Recent human behavioral and neuroimaging results suggest that people are selective in when they encode and retrieve episodic memories. To explain these findings, we trained a memory-augmented neural network to use its episodic memory to support prediction of upcoming states in an environment where past situations sometimes reoccur. We found that the network learned to retrieve selectively as a function of several factors, including its uncertainty about the upcoming state. Additionally, we found that selectively encoding episodic memories at the end of an event (but not mid-event) led to better subsequent prediction performance. In all of these cases, the benefits of selective retrieval and encoding can be explained in terms of reducing the risk of retrieving irrelevant memories. Overall, these modeling results provide a resource-rational account of why episodic retrieval and encoding should be selective and lead to several testable predictions.
Article activity feed
-
-
Evaluation Summary:
This paper addresses an important problem in control of episodic memory. This paper develops a computationally-based proposal about how semantic, working memory, and episodic memory systems might learn to interact so that stored episodic memories can optimally contribute to reconstruction of semantic memory for event sequences. This is an understudied area and this present work can make a major theoretical contribution to this domain with new predictions. The reviewers were positive about the contribution.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)
-
Reviewer #1 (Public Review):
This paper tackles an important problem for cognition and neuroscience. While episodic, semantic, and working memory clearly all interact together in behavior, they have typically been studied independently, and I am unaware of any prior modeling work considering how they might work together so that controlled retrieval of episodes together with semantic processing can support sequential inference/behavior. The proposed model integrates insights from much prior work in a comparatively simple framework; simply getting the systems to work together in a plausible and intuitive way is an impressive accomplishment, and the model's connection to recent imaging results from studies of encoding and retrieval is also promising. For the most part the paper is clearly written, anticipates and addresses important …
Reviewer #1 (Public Review):
This paper tackles an important problem for cognition and neuroscience. While episodic, semantic, and working memory clearly all interact together in behavior, they have typically been studied independently, and I am unaware of any prior modeling work considering how they might work together so that controlled retrieval of episodes together with semantic processing can support sequential inference/behavior. The proposed model integrates insights from much prior work in a comparatively simple framework; simply getting the systems to work together in a plausible and intuitive way is an impressive accomplishment, and the model's connection to recent imaging results from studies of encoding and retrieval is also promising. For the most part the paper is clearly written, anticipates and addresses important questions as it unfolds, and situates both the novel contributions and open questions fairly with respect to the rest of the literature. I think this is an important contribution that will help move the field toward more integrated models of memory and behavior. I offer the following thoughts in hopes of further strengthening a very interesting paper.
I am still not 100% clear on how the sequences were constructed and how these relate to the structure of real schematic events. From the methods I think I understand that the inputs to the model contain a one-hot label indicating the feature observed (e.g. weather), a second one-hot vector indicating the feature value (e.g. rainy), and a third for the query, which indicates what prediction must be made (e.g. "what will the mood be"?). Targets then consist of potential answers to the various queries plus the "I don't know" unit. If this is correct, I was unsure of two things. First, how does the target output (ie the correct prediction) relate to the prior and current features of the event? Like, is the mood always "angry" when the feature is "rainy," or is it chosen randomly for each event, or does it depend on prior states? Does the drink ordered depend on whether the day is a weekday or weekend---in which case the LSTM is obviously critical since this occurs much earlier in the sequence---or does it only depend on the observed features of the current moment (e.g. the homework is a paper), in which case it's less clear why an LSTM is needed. Second, the details of the cover story (going to a coffee shop) didn't help me to resolve these queries; for instance, Figure 2 seems to suggest that the kind of drink ordered depends on the nature of the homework assigned, which doesn't really make sense and makes me question my understanding of Figure 2. In general I think this figure, example, and explanation of the model training/testing sequences could be substantially clarified.
The authors show that their model does a better job of using episodic traces to reinstate the event representation when such traces are stored at the end of an event, rather than when they are stored at both the middle and the end. In explaining why, they show that the model tends to organize its internal representations mainly by time (ie events occurring at a similar point along the sequence are represented as similar). If episodes for the middle of the event are stored, these are preferentially reinstated later as the event continues following distraction, since the start of the "continuing" event looks more like an early-in-time event than a late-in-time event. This analysis is interesting and provides a clear explanation of why the model behaves as it does. However, I wonder if it is mainly due to the highly sequential nature of the events the model is trained on, where prior observed features/queries don't repeat in an event. I wonder if the same phenomenon would be observed richer sequences such as the coffee/tea-making examples from Botvinick et al, where one stirs the coffee twice (once after sugar, once after cream) and so must remember, for each stirring event, whether it is the first or the second. Does the "encode only at the end" phenomenon still persist in such cases? The result is less plausible as an account of the empirical phenomena if it only "works" for strictly linear sequences.
It would be helpful to understand how/why reinforcement learning is preferable to error-correcting learning in this framework. Since the task is simply to predict the upcoming answer to a query given a prior sequence of observed states, it seems likely that plain old backprop could work fine (indeed, the cortical model is pre-trained with backprop already), and that the model could still learn the critical episodic memory gating. Is the reinforcement learning approached used mainly so the model can be connected to the over-arching resource-rational perspective on cognition, or is there some other reason why this approach is preferred?
-
Reviewer #2 (Public Review):
This article presents a network model with multiple regions trained with deep reinforcement learning techniques (Mnih et al. 2016) to address the timing of encoding and retrieval for episodic memory. The authors relate the model results to both behavioral data and functional imaging of the timing of hippocampal-cortical interactions at the transition between events. In many early memory experiments human participants were told when they should encode or retrieve information, but neural models have previously demonstrated the importance of regulating the timing of encoding and retrieval into episodic memory (Hasselmo and Wyble, 1997; Zilli and Hasselmo, 2008). The model presented here focuses on learning of actions that gate encoding of information into episodic memory (EM) in a range of different tasks to be …
Reviewer #2 (Public Review):
This article presents a network model with multiple regions trained with deep reinforcement learning techniques (Mnih et al. 2016) to address the timing of encoding and retrieval for episodic memory. The authors relate the model results to both behavioral data and functional imaging of the timing of hippocampal-cortical interactions at the transition between events. In many early memory experiments human participants were told when they should encode or retrieve information, but neural models have previously demonstrated the importance of regulating the timing of encoding and retrieval into episodic memory (Hasselmo and Wyble, 1997; Zilli and Hasselmo, 2008). The model presented here focuses on learning of actions that gate encoding of information into episodic memory (EM) in a range of different tasks to be used by a neocortical network that has semantic memory in terms of its connectivity and working memory in terms of its ongoing activation, but does not have episodic memory represented in the neocortical circuits. Thus, the neocortical circuit requires gating of episodic memories for effective performance in certain components of behavior tasks. The behavioral tasks being simulated include the task from a recently published task (Chen et al., 2016) in which subjects viewed an episode of the twilight zone in two parts, either on the same day (Recent memory RM) or on two different days (distant memory DM) or only viewed part 2 (no memory). The activation patterns in the model demonstrate that retrieval is effectively used more in the distant memory condition when episodic retrieval from the previous day (rather than working memory from the same day) is necessary for performance of the task. The authors also show that unnecessary episodic retrieval can impede performance by recalling irrelevant information. The results also show interesting effects demonstrating that performance is better when encoding is delayed until the end of an episode, because mid-episode encoding can overshadow the later encoded memory. They effectively link these results on the timing of encoding to data showing activation of hippocampal circuits for episodic encoding at transitions between episodes (Ben-Yakov et al., Reagh et al. 2020). The model addresses important aspects of the control of episodic memory function, and some of its shortcomings are already addressed in the discussion, but there could be further discussion of these points.
Major comments:
1. Line 566 - "new frontier in the cognitive modeling of memory." They should qualify this statement with discussion of the shortcomings of these types of models. This model is useful for illustrating the potential functional utility of controlling the timing of episodic memory encoding and retrieval. However, the neural mechanisms for this control of gating is not made clear in these types of models. The authors suggest a portion of a potential mechanism when they mention the potential influence of the pattern of activity in the decision layer on the episodic memory gating (i.e. depending on the level of certainty - line 325), but they should mention that detailed neural circuit mechanisms are not made clear by this type of continuous firing rate model trained by gradient descent. More discussion of the difficulties of relating these types of neural network models to real biological networks is warranted. This is touched upon in lines 713-728, but the shortcomings of the current model in addressing biological mechanisms are not sufficiently highlighted. In particular, more biologically detailed models have addressed how network dynamics could regulate the levels of acetylcholine to shift dynamics between encoding and retrieval (Hasselmo et al., 1995; Hasselmo and Wyble, 1997). There should be more discussion of this prior work.
2. Figure 1 - "the cortical part of the model" - Line 117 - "cortical weights" and many other references to cortex. The hippocampus is a cortical structure (allocortex), so it is very confusing to see references to cortex used in contrast to hippocampus. The confusing use of references to cortex could be corrected by changing all the uses of "cortex" or "cortical" in this paper to "neocortex" or "neocortical."
3. "the encoding policy is not learned" - This came as a surprise in the discussion, so it indicates that this is not made sufficiently clear earlier in the text. This statement should be made in a couple of points earlier where the encoding policy is discussed. -
-