Increasing stimulus similarity drives nonmonotonic representational change in hippocampus

  1. Jeffrey Wammes
  2. Kenneth A Norman
  3. Nicholas Turk-Browne

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This paper reports a timely, computationally-inspired fMRI analysis of how hippocampus-dependent memory handles overlap in the timing and visual characteristics of objects we encounter. The findings speak to discrepancies in the field over how the hippocampus responds to similarity in memories and will be of broad interest to memory researchers and computational neuroscientists. The elegant experimental approach directly tests the predictions of a theoretical framework by parametrically manipulating visual overlap between associated stimuli. The analyses are clearly reported and directly address the hypothesis. The conclusions are sound. However, these findings may not yet generalize beyond visual similarity in the context of temporal co-occurrence or statistical learning, and some concern is raised over the theoretical groundwork for hippocampal subregion predictions and how context and overlap are considered in their memory network model.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Studies of hippocampal learning have obtained seemingly contradictory results, with manipulations that increase coactivation of memories sometimes leading to differentiation of these memories, but sometimes not. These results could potentially be reconciled using the nonmonotonic plasticity hypothesis, which posits that representational change (memories moving apart or together) is a U-shaped function of the coactivation of these memories during learning. Testing this hypothesis requires manipulating coactivation over a wide enough range to reveal the full U-shape. To accomplish this, we used a novel neural network image synthesis procedure to create pairs of stimuli that varied parametrically in their similarity in high-level visual regions that provide input to the hippocampus. Sequences of these pairs were shown to human participants during high-resolution fMRI. As predicted, learning changed the representations of paired images in the dentate gyrus as a U-shaped function of image similarity, with neural differentiation occurring only for moderately similar images.

Article activity feed

  1. Version published to 10.7554/elife.68344 on eLife
  2. eLife

    Author Response:

    Reviewer #1:

    In this paper, Wammes et al. used fMRI to investigate changes in representational similarity of temporally paired images in hippocampal subfields. The stimuli were designed to parametrically vary in their visual similarity so that individual pairs covered the entire range of visual overlap, which was behaviourally validated by a separate sample of participants. The authors compared the neural patterns evoked by these pairs of stimuli before and after participants completed a statistical learning task. The findings showed that pre- to post-learning, representations in the dentate gyrus reconfigured to fit a cubic model, consistent with the non-monotonic plasticity hypothesis (NMPH).

    This is an interesting, novel approach with a clever stimulus manipulation which addresses a gap in the current literature. The study is well-motivated by theory, the analyses are appropriate and clearly described, the implemented controls are carefully designed, and the manuscript is well-written. However, it is unclear whether the same principles necessarily generalize beyond visual similarity, and whether these neural patterns meaningfully relate to behaviour.

    1. The analytic approach is well-designed and the results clearly address the hypotheses. However, it seems like the conclusions might be dependent on this learning paradigm, which should be discussed in a bit more detail and made clearer. The present statistical learning approach is somewhat implicit in its nature and relies on the participants gradually recognizing the temporal links between stimuli. In contrast, in most prior studies cited in the present manuscript, participants were explicitly instructed to make associations between stimuli that either occurred on the screen simultaneously, or relatively far apart in time (i.e., not successively). This top-down influence likely plays an important role. Even beyond experimental paradigms - we often make connections between similar experiences that occurred far apart in time, and cannot always rely on temporal contingencies. The step between previous work and statistical learning needs to be made clearer and more explicit.

    Although our current approach involves a more implicit statistical learning task, the hypothesized non-monotonic plasticity is a general mechanism that has been and can be applied across tasks. We used temporal contingency to create a situation where representations were concurrently active. However, prior work has used other manipulations, such as linking to a shared associate. We have modified and expanded both the Introduction and Conclusion to emphasize this broader context and highlight directions for future work.

    See Introduction (p. 4, lines 60-74): “The NMPH has been put forward as a learning mechanism that applies broadly across tasks in which memories compete, whether they have been linked based on incidental co-occurrence in time or through more intentional associative learning (Ritvo et al., 2019). The NMPH can explain findings of differentiation in diverse paradigms (e.g., linking to a shared associate: Chanales et al., 2017; Favila et al., 2016; Schlichting et al., 2015; Molitor et al., 2020; retrieval practice: Hulbert & Norman, 2015; statistical learning: Kim, Norman, & Turk-Browne, 2017) by positing that these paradigms induced moderate coactivation of competing memories. Likewise, relying on the same parameter of coactivation, the NMPH can explain seemingly contradictory findings showing that shared associates (Collin et al., 2015; Milivojevic et al., 2015; Schlichting et al., 2015; Molitor et al., 2020) and co-occurring items (Schapiro et al., 2012; Schapiro, Turk-Browne, Norman, & Botvinick, 2016) can lead to integration by positing that — in these cases — the paradigms induced strong coactivation. Importantly, although the NMPH is compatible with findings of both differentiation and integration across several paradigms with diverse task demands, the explanations above are post hoc and do not provide a principled test of the NMPH’s core claim that there is a continuous, U-shaped function relating the level of coactivation to representational change.

    See Introduction (p. 5, lines 83-86): “No existing study has demonstrated the full U- shaped pattern for representational change; that is what we set out to do here, using a visual statistical learning paradigm — specifically, we brought about coactivation using temporal co-occurrence between paired items, and we manipulated the degree of coactivation by varying the visual similarity of the items in a pair.”

    See Conclusion (p. 18, lines 370-374): “From a theoretical perspective, these results provide the strongest evidence to date for the NMPH account of hippocampal plasticity. We expect that a similar U-shaped function relating coactivation and representational change will manifest in paradigms with different task demands and stimuli, but additional work is needed to provide empirical support for this claim about generality.”

    1. Related to the point above - the timecourse over which such statistical learning occurs should be discussed. If I understood correctly, all of the learning occurred in the 6 scanned blocks between the two templating runs. Does the NMPH predict that the hippocampal patterns should immediately reconfigure depending on visual input, or only reconfigure once the participants encode the links between paired stimuli? If the pattern consistent with the NMPH is immediately evident, this would suggest that the present findings, while very convincing, might not be governed by the same mechanisms as integration/differentiation in memory. It seems unlikely that participants would immediately attempt to link these complex visual stimuli, especially as the cover task was orthogonal. To this end, it would be helpful to see any kind of analysis evaluating representations across the 6 statistical learning runs.

    The reviewer correctly describes that learning took place over the six blocks between templating runs. We agree that observing the emergence of representational change across those runs would be ideal. Unfortunately, however, our design is not compatible with this analysis. Because the pairs were learned from deterministic transition probabilities, the onsets of the paired stimuli were correlated in time. When these correlated events are convolved with the slow hemodynamic response, the responses to the paired stimuli cannot be reliably distinguished. Also, the response to the second stimulus in a pair would be affected by visual similarity to its preceding stimulus as a result of adaptation/repetition suppression, confounding comparisons across conditions. These problems are precisely why we employed a pre/post design in which to-be/previously paired stimuli are presented independently in a random order. This allows for the assessment of representational similarity unconfounded with correlated onsets or adaptation.

    Although we cannot provide a sense of the learning trajectory, we now highlight this design decision, acknowledge the limitation, and highlight this as an opportunity for future work with other more time-resolved modalities or with (random) representational assessments interdigitated with the learning blocks.

    See Discussion (p. 17, lines 358-366): “Finally, although analyzing representational overlap in templating runs before and after statistical learning afforded us the ability to quantify pre-to-post changes, our design precluded analysis of the emergence of representational change over time. That is, we could not establish whether integration or differentiation occurred early or late in statistical learning. This is because, during statistical learning runs, the onsets of paired images were almost perfectly correlated, meaning that it was not possible to distinguish the representation of one image from its pairmate. Future work could monitor the time course of representational change, either by interleaving additional templating runs throughout statistical learning (although this could interfere with the statistical learning process), or by exploiting methods with higher temporal resolution where the responses to stimuli presented close in time can more readily be disentangled.”

    1. In the Introduction and Discussion, the authors focus on learning and discuss the integration/differentiation of memories. To establish a link between the reported hippocampal representations and behaviour, it would be helpful to show evidence of a link between neural differentiation and measures of statistical learning such as priming.

    As the reviewer alluded to earlier, our behavioral task is orthogonal to the manipulation of temporal co-occurrence. Accordingly, we do not have any behavioral data on which we could conduct such an analysis. We fully acknowledge the value of this suggestion and now describe this as a limitation and area for future research.

    See Discussion (p. 17, lines 350-357): “Prior work in this area has demonstrated brain- behavior relationships (Favila et al., 2016; Molitor et al., 2020), so it is clear that changes in representational overlap (i.e., either integration or differentiation) can bear on later behavioral performance. However, in the current work, our behavioral task was intentionally orthogonal to the dimensions of interest (i.e., unrelated to temporal co- occurrence and visual similarity), limiting our ability to draw conclusions about potential downstream effects on behavior. We believe that this presents a compelling target for follow-up research. Establishing a behavioral signature of both integration and differentiation in the context of nonmonotonic plasticity will not only clarify the brain-behavior relationship, but also allow for investigations in this domain without requiring brain data.”

    1. From the authors' predictions (and Fig 1), it might follow that participants who show steeper slopes in early visual regions (i.e., higher correspondence to stimulus similarity) pre-learning might also show a stronger cubic trend in the hippocampus. It would be useful to show within-participant analyses to link visual processing regions to hippocampal representations.

    What a fantastic suggestion! To test this prediction, we extracted the linear coefficients in the visual similarity analysis from cortical ROIs (V1, V2, LO, IT, FG, PHC, PRC, and EC) and the cubic model fit in the representational change analysis from the key hippocampal ROI (DG). Linearity during the initial templating run in PRC was associated with stronger non-monotonicity in DG. The full reporting of these analyses is now included in the figure supplements and referenced in the main text.

    See Results, subsection Representational Change (p. 12, lines 228-229): “Interestingly, in an exploratory analysis, we found that the degree of model fit in DG was predicted by the extent to which visual representations in PRC tracked model similarity (see Figure 4—figure supplement 2).”

    Reviewer #2:

    The authors apply neural network modeling and representational analysis of fMRI data to testing the ability of the theoretical framework under the "non-monotonic-plasticity hypothesis" to explain how hippocampal subdivisions represent similarity and distinctiveness between events. They suggest that the dentate gyrus subfield, in particular, was sensitive to the degree of overlap between experiences, and changes how it favored distinctiveness or similarity in its representation of associated stimuli in a non-monotonic manner.

    Overall, the work builds logically on prior evidence from this group focused on how cortical representations influence memory, and leverages a compelling theoretical framework to reconcile some conflict in the literature on how hippocampal representations respond to overlap.

    The primary confusion and concern with the current manuscript was on the theoretical side. It was not wholly clear from the literature review why DG was the predicted locus of the non-monotonic representational relationship observed, and how the findings fit with extant data from rodent work.

    Thank you for providing an opportunity to better motivate our work. We have updated the paragraph justifying our focus on the hippocampus and on DG in particular.

    See Introduction (p. 8, lines 122-147): “We and others have previously hypothesized that nonmonotonic plasticity applies widely throughout the brain (Ritvo et al., 2019), including sensory regions (e.g., Bear, 2003). In this study, we focused on the hippocampus because of its well-established role in supporting learning effects over relatively short timescales (e.g., Favila et al., 2016; Kim et al., 2017; Schapiro et al., 2012). Importantly, we hypothesized that, even if nonmonotonic plasticity occurs throughout the entire hippocampus, it might be easier to trace out the full predicted U-shape in some hippocampal subfields than in others. As discussed above, our hypothesis is that representational change is determined by the level of coactivation — detecting the U-shape requires sweeping across the full range of coactivation values, and it is particularly important to sample from the low-to-moderate range of coactivation values associated with the differentiation ‘dip’ in the U-shaped curve (i.e., the leftmost side of the inset in Fig. 1). Prior work has shown that there is extensive variation in overall activity (sparsity) levels across hippocampal subfields, with CA2/3 and DG showing much sparser codes than CA1 (Barnes, McNaughton, Mizumori, Leonard, & Lin, 1990; Duncan & Schlichting, 2018). We hypothesized that regions with sparser levels of overall activity (DG, CA2/3) would show lower overall levels of coactivation and thus do a better job of sampling this differentiation dip, leading to a more robust estimate of the U-shape, compared to regions like CA1 that are less sparse and thus should show higher levels of coactivation (Ritvo et al., 2019). Consistent with this idea, human fMRI studies have found that CA1 is relatively biased toward integration and CA2/3/DG are relatively biased toward differentiation (Dimsdale-Zucker et al., 2018; Kim et al., 2017; Molitor et al., 2020). Zooming in on the regions that have shown differentiation in human fMRI (CA2/3/DG), we hypothesized that the U-shape would be most visible in DG, for two reasons: First, DG shows sparser activity than CA3 (Barnes et al., 1990; GoodSmith et al., 2017; West, Slomianka, & Gundersen, 1991) and thus will do a better job of sampling the left side of the coactivation curve. Second, CA3 is known to show strong attractor dynamics (‘pattern completion’; McNaughton & Morris, 1987; Rolls & Treves, 1998; Guzowski, Knierim, & Moser, 2004) that might make it difficult to observe moderate levels of coactivation. For example, rodent studies have demonstrated that, rather than coactivating representations of different locations, CA3 patterns tend to sharply flip between one pattern and the other (e.g., Leutgeb, Leutgeb, Moser, & Moser, 2007; Vazdarjanova & Guzowski, 2004).”

    Additionally, the theoretical model (nicely illustrated in the manuscript) is considered in a somewhat biological-network-agnostic level. Some assumption for how context changes over time, how prior representations are maintained over time, etc., are important for non-monotonic relationships between representations and memory to manifest in the model, but the manuscript does not provide much discussion of their plausibility. This was particularly notable in terms of the emphasis given in the fMRI data to different hippocampal subfields, but not much discussion given on whether/why the model framework is static across subfields (in terms of how context and item information are represented and connected).

    We appreciate this nudge to discuss these additional subfield-specific factors; we have added a paragraph to the Discussion that addresses these issues.

    See Discussion (p. 16, lines 318-336): “Although we focused above on differences in sparsity when motivating our predictions about subfield-specific learning effects, there are numerous other factors besides sparsity that could affect coactivation and (through this) modulate learning. For example, the degree of coactivation during statistical learning will be affected by the amount of residual activity of the A item during the B item’s presentation in the statistical learning phase. In Figure 1, this residual activity is driven by sustained firing in cortex, but this could also be driven by sustained firing in hippocampus; subfields might differ in the degree to which activation of stimulus information is sustained over time (see, e.g., the literature on hippocampal time cells: Eichenbaum, 2014; Howard & Eichenbaum, 2013), and activation could be influenced by differences in the strength of attractor dynamics within subfields (e.g., Neunuebel & Knierim, 2014; Leutgeb et al., 2007). Also, in Figure 1, the learning responsible for differentiation was shown as happening between ‘perceptual conjunction’ neurons and ‘context’ neurons in the hippocampus. Subfields may vary in how strongly these item and context features are represented, in the stability/drift of the context representations (DuBrow, Rouhani, Niv, & Norman, 2017), and in the interconnectivity between item and context features (Witter, Wouterlood, Naber, & Van Haeften, 2000). It is also likely that some of the relevant plasticity between item and context features happens across, in addition to within, subfields (Hasselmo & Eichenbaum, 2005). For these reasons, exploring the predictions of the NMPH in the context of biologically detailed computational models of the hippocampus (e.g., Schapiro, Turk-Browne, Botvinick, & Norman, 2017; Frank, Montemurro, & Montaldi, 2020; Hasselmo & Wyble, 1997) will help to sharpen predictions about what kinds of learning should occur in different parts of the hippocampus.

    As such, this review was very positive, and found the methods to be sound and the conclusions to be solid. There was some room for improvement in how the theoretical foundation was presented for the hippocampal subregion fMRI predictions and for the conceptualization of the neural network memory model.

    We agree with the reviewer that more justification of our specific hippocampal predictions was required and we are grateful for their suggestions.

  3. eLife

    Reviewer #2 (Public Review):

    The authors apply neural network modeling and representational analysis of fMRI data to testing the ability of the theoretical framework under the "non-monotonic-plasticity hypothesis" to explain how hippocampal subdivisions represent similarity and distinctiveness between events. They suggest that the dentate gyrus subfield, in particular, was sensitive to the degree of overlap between experiences, and changes how it favored distinctiveness or similarity in its representation of associated stimuli in a non-monotonic manner.

    Overall, the work builds logically on prior evidence from this group focused on how cortical representations influence memory, and leverages a compelling theoretical framework to reconcile some conflict in the literature on how hippocampal representations respond to overlap.

    The primary confusion and concern with the current manuscript was on the theoretical side. It was not wholly clear from the literature review why DG was the predicted locus of the non-monotonic representational relationship observed, and how the findings fit with extant data from rodent work.

    Additionally, the theoretical model (nicely illustrated in the manuscript) is considered in a somewhat biological-network-agnostic level. Some assumption for how context changes over time, how prior representations are maintained over time, etc., are important for non-monotonic relationships between representations and memory to manifest in the model, but the manuscript does not provide much discussion of their plausibility. This was particularly notable in terms of the emphasis given in the fMRI data to different hippocampal subfields, but not much discussion given on whether/why the model framework is static across subfields (in terms of how context and item information are represented and connected).

    As such, this review was very positive, and found the methods to be sound and the conclusions to be solid. There was some room for improvement in how the theoretical foundation was presented for the hippocampal subregion fMRI predictions and for the conceptualization of the neural network memory model.

  4. eLife

    Reviewer #1 (Public Review):

    In this paper, Wammes et al. used fMRI to investigate changes in representational similarity of temporally paired images in hippocampal subfields. The stimuli were designed to parametrically vary in their visual similarity so that individual pairs covered the entire range of visual overlap, which was behaviourally validated by a separate sample of participants. The authors compared the neural patterns evoked by these pairs of stimuli before and after participants completed a statistical learning task. The findings showed that pre- to post-learning, representations in the dentate gyrus reconfigured to fit a cubic model, consistent with the non-monotonic plasticity hypothesis (NMPH).

    This is an interesting, novel approach with a clever stimulus manipulation which addresses a gap in the current literature. The study is well-motivated by theory, the analyses are appropriate and clearly described, the implemented controls are carefully designed, and the manuscript is well-written. However, it is unclear whether the same principles necessarily generalize beyond visual similarity, and whether these neural patterns meaningfully relate to behaviour.

    1. The analytic approach is well-designed and the results clearly address the hypotheses. However, it seems like the conclusions might be dependent on this learning paradigm, which should be discussed in a bit more detail and made clearer. The present statistical learning approach is somewhat implicit in its nature and relies on the participants gradually recognizing the temporal links between stimuli. In contrast, in most prior studies cited in the present manuscript, participants were explicitly instructed to make associations between stimuli that either occurred on the screen simultaneously, or relatively far apart in time (i.e., not successively). This top-down influence likely plays an important role. Even beyond experimental paradigms - we often make connections between similar experiences that occurred far apart in time, and cannot always rely on temporal contingencies. The step between previous work and statistical learning needs to be made clearer and more explicit.

    2. Related to the point above - the timecourse over which such statistical learning occurs should be discussed. If I understood correctly, all of the learning occurred in the 6 scanned blocks between the two templating runs. Does the NMPH predict that the hippocampal patterns should immediately reconfigure depending on visual input, or only reconfigure once the participants encode the links between paired stimuli? If the pattern consistent with the NMPH is immediately evident, this would suggest that the present findings, while very convincing, might not be governed by the same mechanisms as integration/differentiation in memory. It seems unlikely that participants would immediately attempt to link these complex visual stimuli, especially as the cover task was orthogonal. To this end, it would be helpful to see any kind of analysis evaluating representations across the 6 statistical learning runs.

    3. In the Introduction and Discussion, the authors focus on learning and discuss the integration/differentiation of memories. To establish a link between the reported hippocampal representations and behaviour, it would be helpful to show evidence of a link between neural differentiation and measures of statistical learning such as priming.

    4. From the authors' predictions (and Fig 1), it might follow that participants who show steeper slopes in early visual regions (i.e., higher correspondence to stimulus similarity) pre-learning might also show a stronger cubic trend in the hippocampus. It would be useful to show within-participant analyses to link visual processing regions to hippocampal representations.

  5. eLife

    Evaluation Summary:

    This paper reports a timely, computationally-inspired fMRI analysis of how hippocampus-dependent memory handles overlap in the timing and visual characteristics of objects we encounter. The findings speak to discrepancies in the field over how the hippocampus responds to similarity in memories and will be of broad interest to memory researchers and computational neuroscientists. The elegant experimental approach directly tests the predictions of a theoretical framework by parametrically manipulating visual overlap between associated stimuli. The analyses are clearly reported and directly address the hypothesis. The conclusions are sound. However, these findings may not yet generalize beyond visual similarity in the context of temporal co-occurrence or statistical learning, and some concern is raised over the theoretical groundwork for hippocampal subregion predictions and how context and overlap are considered in their memory network model.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)

  6. Version published to 10.1101/2021.03.13.435275v1 on bioRxiv