Neural learning rules for generating flexible predictions and computing the successor representation

  1. Ching Fang
  2. Dmitriy Aronov
  3. LF Abbott
  4. Emily L Mackevicius

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This important work provides compelling evidence for the biological plausibility of the Successor Representation (SR) algorithm. The SR is a leading computational hypothesis to explore whether neural representations are consistent with the hypothesis that the neural networks in specific brain area perform predictive computations. Establishing a biologically plausible learning rule for SR representations to form is of high importance in the field of neuroscience. This is also important for comparing the predictive ability of neural circuits with other predictive frameworks designed in machine learning.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

The predictive nature of the hippocampus is thought to be useful for memory-guided cognitive behaviors. Inspired by the reinforcement learning literature, this notion has been formalized as a predictive map called the successor representation (SR). The SR captures a number of observations about hippocampal activity. However, the algorithm does not provide a neural mechanism for how such representations arise. Here, we show the dynamics of a recurrent neural network naturally calculate the SR when the synaptic weights match the transition probability matrix. Interestingly, the predictive horizon can be flexibly modulated simply by changing the network gain. We derive simple, biologically plausible learning rules to learn the SR in a recurrent network. We test our model with realistic inputs and match hippocampal data recorded during random foraging. Taken together, our results suggest that the SR is more accessible in neural circuits than previously thought and can support a broad range of cognitive functions.

Article activity feed

  1. Version published to 10.7554/elife.80680 on eLife
  2. eLife

    Evaluation Summary:

    This important work provides compelling evidence for the biological plausibility of the Successor Representation (SR) algorithm. The SR is a leading computational hypothesis to explore whether neural representations are consistent with the hypothesis that the neural networks in specific brain area perform predictive computations. Establishing a biologically plausible learning rule for SR representations to form is of high importance in the field of neuroscience. This is also important for comparing the predictive ability of neural circuits with other predictive frameworks designed in machine learning.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their name with the authors.)

  3. eLife

    Reviewer #1 (Public Review):

    The authors aim at establishing a biologically plausible learning rule for the Successor Representation (SR) to be computed by neural circuits.

    The study is well designed with a strong logical flow moving from a simple example (random process on a circle) to comparison with real neural data. The manuscript is well written in all its components and figures are clear. All the results provided in the main paper are backed up by a thorough theoretical analysis outlined in the supplementary material. As it is common the theoretical analysis does not have much space in the manuscript. I would suggest summarizing with more specific statements the theoretical results that are achieved whenever there is a pointer to a supplementary note.

    While the authors perform an extensive and careful review of the literature, a lot of it is confined to the Discussion. As the results of the paper strongly rely on the normalizing term in Eq.4. I would suggest potentially moving upfront part of the discussion of this term. I would suggest enlarging the paragraph that discusses the biological plausibility of this specific term. Clearly laying out, for the non-expert reader, why it is biologically plausible compared to other learning rules. And I would also consider moving the required material to establish the novelty of such term: a targeted review of the relevant literature (current lines 358-366 and 413-433). This would allow the reader to understand immediately the significance and relative novelty of such term. For example, I personally wondered while reading the paper how different was such term from the basic idea of Fiete et al. Neuron 2010 (DOI 10.1016/j.neuron.2010.02.003).

    I would also suggest writing a "limitations" paragraph in the discussion clearly outlining what this learning rule couldn't achieve. For example, Stachenfeld et al Nat.Neuro. have many examples where the SR is deployed. I wonder if the learning rule suggested by the authors would always work across the board, or if there are limitations that could be highlighted where the framework suggested would not work well. I am not suggesting performing more experiments/simulations but simply sharing insight regarding the results and the capability of the proposed learning rule.

  4. eLife

    Reviewer #2 (Public Review):

    The paper presents a novel and biologically motivated method for computing the successor representation. Given the line of research suggesting that the hippocampus computes successor representations, it is a valuable contribution to develop such a model which more closely matches known hippocampal dynamics. In this case, the recurrent dynamics of the hippocampus are used as inspiration for a recurrent neural network based model. This proposed method is capable of rapidly learning the SR for a given environment and behavioral policy, overcoming weaknesses of previous methods.

    To demonstrate the performance of their method, the authors compare their model to a traditional linear successor representation algorithm, and demonstrate that their method is able to learn the SR more rapidly for simple environments. They also compare the ability of their method and the baseline to model real neural data recorded from a foraging animal. Here they demonstrate that their method better matches the place cell representations found in the neural data than the baseline model.

    Much of the results section of the paper is devoted to a description of the various design choices made in the author's model, providing a step by step analysis of each feature of the model. These design choices each seem well founded, and suggest a deep understanding of the underlying dynamics of the system being studied. While it is important to understand how and why the model functions the way it does, some of this information may be unnecessary for inclusion in the primary results section, and detracts from the flow of the paper.

    As a consequence of the body of the text being devoted to the analysis of the design choices behind the proposed model, a relatively smaller portion of the work involves direct comparisons with neural data. In these comparisons, while it is apparent that there is a reasonable match between the proposed model and the empirical data, it is difficult to interpret these results. This is because it is unclear what should be expected of a good or bad model given the data being analyzed (TD error and KL divergence), and reasonable baselines to compare against are not presented outside of the traditional TD algorithm, which is shown to be comparable to the proposed RNN based method in a number of cases.

  5. eLife

    Reviewer #3 (Public Review):

    Experimental and computational works have proposed that neurons in the hippocampus represent a predictive map of the environment called the successor representation. This theoretical study examines how plasticity in a model network of recurrently connected neurons can lead to such a representation. The main conclusion is that any plasticity rule that encodes transition probabilities in synaptic weights gives rise to the successor representation at the level of neural activity. This fundamental theoretical insight gives additional credibility to the idea that the hippocampus can implement the successor representation.

    Strengths:
    - elegantly designed theoretical study
    - very clear writing that progressively introduces the main result and argues for its generality
    - comparison of the model with data in a random foraging task

    Weaknesses:
    - certain technical choices need additional motivation

  6. Version published to 10.1101/2022.05.18.492543v1 on bioRxiv