Contribution of behavioural variability to representational drift

  1. Sadra Sadeh
  2. Claudia Clopath

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This work builds on rapidly accumulating evidence for the importance of measuring and accounting for behavior in neural data, and will be of interest to a broad neuroscience audience. Analyses of Allen Brain Atlas datasets show that sensory representations change and match up reliably with behavioral state. The manuscript's main conclusions are supported by the data and analyses and the work raises important questions about previous accounts of the sources of representational drift in sensory areas of the brain.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Neuronal responses to similar stimuli change dynamically over time, raising the question of how internal representations can provide a stable substrate for neural coding. Recent work has suggested a large degree of drift in neural representations even in sensory cortices, which are believed to store stable representations of the external world. While the drift of these representations is mostly characterized in relation to external stimuli, the behavioural state of the animal (for instance, the level of arousal) is also known to strongly modulate the neural activity. We therefore asked how the variability of such modulatory mechanisms can contribute to representational changes. We analysed large-scale recording of neural activity from the Allen Brain Observatory, which was used before to document representational drift in the mouse visual cortex. We found that, within these datasets, behavioural variability significantly contributes to representational changes. This effect was broadcasted across various cortical areas in the mouse, including the primary visual cortex, higher order visual areas, and even regions not primarily linked to vision like hippocampus. Our computational modelling suggests that these results are consistent with independent modulation of neural activity by behaviour over slower timescales. Importantly, our analysis suggests that reliable but variable modulation of neural representations by behaviour can be misinterpreted as representational drift if neuronal representations are only characterized in the stimulus space and marginalized over behavioural parameters.

Article activity feed

  1. Version published to 10.7554/elife.77907 on eLife
  2. Version published to 10.1101/2022.01.02.474731v2 on bioRxiv
  3. eLife

    Author Response

    Reviewer 1

    Sadeh and Clopath analyze two mouse datasets from the Allen Brain Atlas and show that sensory representations can have apparent representational drift that is entirely due to behavioral modulation. The analysis serves as a caution against over-interpreting shifts in the neural code. The analysis of data is coupled with careful modeling work that shows that the behavioral state reliably shifts sensory representations independently of stimulus modulation (rather than acting as a gain factor), and further show that it is reproducibly shifted when the behavioral state is adequately controlled for. The methods presented point towards a more careful consideration and measurement of behavioral states during sensory recordings, and a re-analysis of previous findings. The findings held up for both standard drifting grating stimuli as well as natural movies.

    The fact that neurons may have different tuning depending on the behavioral state of the animal raises obvious questions about readout. The authors show that neurons with strong behavioral shifts should simply be ignored and that this can be achieved if the downstream decoder weights inputs with more stimulus information. While questions remain about why behavior shifts representations and how that could be more effectively utilized by downstream circuits, the results presented clearly show that sensory representations might not always be simply drifting over time, and will spark some careful analysis of past and future experimental results.

    Many thanks for a clear summary of the work and emphasizing the significance of the results.

    Reviewer 2

    Studies from recent years have shown that neuronal responses to the same stimuli or behavior can gradually change with time - a phenomenon known as representational drift. Other recent studies have shown that changes in behavior can also modulate neuronal responses to a given sensory stimulus. In this manuscript, Sadeh and Clopath analyzed publicly available data from the Allen Institute to examine the relationship between animal behavioral variability and changes in neuronal representations. The paper is timely and certainly has the potential to be of interest to neuroscientists working in different fields. However, there are currently several important issues with the analysis of the data and their interpretations that the authors should address. We believe that after these concerns are addressed, this study will be an important contribution to the field.

    We really appreciate the time and the effort the reviewer(s) have taken to evaluate our results and analysis in detail. Their comments are very relevant and critical to the improvement of the manuscript. We explain below how we addressed their various comments and concerns

    1. The manuscript raises a potential problem: while previous work suggested that the passage of time leads to gradual changes in neuronal responses, the causality structure is different: i.e., the passage of time leads to gradual changes in behavior, which in turn lead to gradual changes in neuronal responses. The authors conclude that "variable behavioral signal might be misinterpreted as representational drift". While this may be true, in its current form, the paper lacks critical analyses that would support such a claim. It is possible that both factors - time and behavior - have a unique contribution to changes in neuronal responses, or that only time elicits changes in neuronal responses (and behavior is just correlated with time). Thus, the authors should demonstrate that these changes cannot be explained solely by the passage of time and elucidate the unique contributions of behavior (and elapsed time) to changes in representations.

    This is a very important point and we addressed it with new analyses, by dedicating a new figure (Figure 1–figure supplement 5) and a new part of the Results section to it. The results of our new analyses show that strong representational drift mainly exists in those animals/sessions with large behavioral changes between the two blocks, and that in animals/sessions with small behavioral changes, such drift is minimal, despite the passage of time (see our responses below to Major comments for further details).

    1. There are also several issues with the analysis of the data and the presentation of the results. The most concerning of which is that the data shows a non-linear (and non-monotonic) relationship between behavioral changes and representational similarity. In many of the presented cases, the data points fall into two or more discrete clusters. This can lead to the false impression that there is a monotonic relationship between the two variables, even though there is no (or even opposite) relationship within each cluster. This is a crucial point since the clusters of data points most likely represent different blocks that were separated in time (or separation between within-block and acrossblock comparisons).

    This is an important concern. To address this, we analyzed the source of the non-monotonic relationship / opposite trend in the data and demonstrated the results in a new figure (Figure 4–figure supplement 2). Our results show that the non-monotonic relationship does not compromise the result of our previous analysis. Furthermore, it suggests that the non-monotonic / opposite trend is emerging as a result of more complex interactions between different aspects of behavior. We have also shown, in separate analyses, that the passage of time is not the main contributing factor to representational drift, rather large behavioral changes are correlated with strong drifts between the two blocks of presentation (Figure 1—figure supplement 5, and Figure 3—figure supplement 2).

    More generally, we did not intend to claim that the relationship with behavioral changes is linear or/and monotonic. We used linear analysis just to show the main trend of decrease in representational similarity with large behavioral changes. Any other analysis should assume some form of nonlinearity, but because the nonlinear relationships between behavior and activity were complex, it was not easy to assume such nonlinearity.

    We in fact tried to use two other ways of analysis, nonlinear correlations and generalized linear models (GLM), but there were issues hindering a proper use of each analysis. Nonlinear correlations assume a specific type of nonlinearity, but the nature of nonlinearity underlying the data is not clear (in fact, it looks to be different in different example non-monotonic trends in the data). We could not, therefore, assume a nonlinearity that best fitted all the data; we believe the nature of this nonlinearity, or how behavior modulates neuronal activity in a nonlinear manner, is in itself an interesting and open question for future investigation, but beyond the scope of this study. GLM did not provide useful results either, as the relationship between behavioral changes and neural activity/representational similarity was state-dependent and transitioning between nonlinear states, therefore hindering the usage of linear methods.

    We therefore opted for the simplest analysis which can show and quantify this dependence - emphasizing that further analyses are in fact needed to get to the bottom of the exact nonlinear relationship (for further details, see the responses below to Major comments).

    1. The authors also suggest that using measures of coding stability such as 'population-vector correlations' may be problematic for quantifying representational drift because it could be influenced by changes in the neuronal activity rates, which may be unrelated to the stimulus. We agree that it is important to carefully dissociate between the effects of behavior on changes in neuronal activity that are stimulus-dependent or independent, but we feel that the criticism raised by the authors ignores the findings of multiple previous papers, which (1) did not purely attribute the observed changes to the sensory component, and (2) did dissociate between stimulus-dependent changes (in the cells' tuning) and off-context/stimulus-independent changes (in the cells' activity rates).

    That’s a very valid point. As population vector correlations are used quite often in (experimental and theoretical) works on representational drift, we wanted to highlight the pitfalls of such a metric in dissociating between sensory-evoked and sensory-independent components. However, as the reviewers have mentioned, these two aspects have been separated and addressed independently in some of the past literature in the field. For instance, as we discussed in the Discussion, Deitch et al. (Current Biology, 2021) have calculated this for different metrics, including tuning curve correlations, which can potentially alleviate this problem:

    “A recent analysis of similar datasets from the Allen Brain Observatory reported similar levels of representational drift within a day and over several days5. The study showed that tuning curve correlations between different repeats of the natural movies were much lower than population vector and ensemble rate correlations5; it would be interesting to see if, and to which extent, similarity of population vectors due to behavioural signal that we observed here may contribute to this difference.”

    We tried to highlight these contributions better in the revised manuscript (see further on this below in our responses to Major comments).

    1. Another important issue relates to the interchangeable use of the terms 'representational drift' and 'representational similarity'. Representational similarity is a measure to identify changes in representations, and drift is one such change. This may confuse the reader and lead to the misconception that all changes in neuronal responses are representational drift.

    We thank the reviewer(s) for raising this point. We have clarified our use of the terms representational similarity and representational drift in the revised manuscript. Specifically, we have quantified representational drift index between the two blocks according to a previously used metric (RDI; Marks & Goard, 2021) in our new analysis (Figure 1–figure supplement 5).

    For the main part of the paper, however, we have decided to base our analysis on representational similarity (RS), and to evaluate the drop of RS with changes in behavior. Our reasoning for this is twofold. First, any measure of representational drift should ultimately be a function of the representational similarity. The measure we used above, for instance, is calculated as RD = (RS_ws - RS_bs)/(RS_ws + RS_bs) (Marks and Goddard, 2021), with RS_ws and RS_bs referring to the average representational similarity within a session or between different sessions. However, RS contains more information, especially with regard to fine-tuned changes - the above metric, for instance, averages all the changes within each block of presentation. By focusing on the basic function of representational similarity, we could capture both the gross changes between the blocks as well as more nuanced changes that can arise within them, especially with regard to behavioral changes. Another aspect that would have been lost by only using the usual metric of representational drift is the direction of change. In our analysis, we in fact found that the average RS increased within the second block of presentation, which might be contrary to the usual direction of drift. We found this unconventional change of RS interesting and informative too. We could highlight that, presenting the raw RS provided a better analysis strategy. Based on these reasons, we think representational similarity would be a better metric to base our analyses upon, although we have now calculated a conventional representational drift index for comparison too.

    Reviewer 3

    Although it is increasingly realized that cortical neural representations are inherently unstable, the meaning of such "drift" can be difficult or impossible to interpret without knowing how the representations are being read out and used by the nervous system (i.e. how it contributes to what the experimental animal is actually doing now or in the future). Previous studies of representational drift have either ignored or explicitly rejected the contribution of what the animal is doing, mostly due to a lack of high-dimensional behavioural data. Here the authors use perhaps the most extensive opensource and rigorous neural data available to take a more detailed look at how behaviour affects cortical neural representations as they change over repeated presentations of the same visual stimuli.

    The authors apply a variety of analyses to the same two datasets, all of which convincingly point to behavioural measures having a large impact on changing neural representations. They also pit models against each other to address how behavioural and stimulus signals combine to influence representations, whether independently or through behaviour influencing the gain of stimuli. One analysis uses subsets of neurons to decode the stimulus, and the independent model correctly predicts the subset to use for better decoding. However, one caveat may be that the nervous system does not need to decode the stimulus from the cortex independently of behaviour; if necessary, this could be done elsewhere in the nervous system with a parallel stream of visual information.

    Overall the authors' claims are well-supported and this study should lead to a re-assessment of the concept of "representational drift". Nonetheless, a weakness of all analyses presented here is that they are all based on data in head-fixed mice that were passively viewing visual stimuli, such that it is unclear what relevance the behaviour has. Furthermore, the behavioural measurements available in the opensource dataset (pupil movements and running speed) are still a very low dimensional representation of what the mice were actually doing (e.g. detailed kinematics of all body movements and autonomic outputs). Thus, although the authors here as well as other large-scale neural recording studies in the past decade or so make it clear that relatively basic measures of behaviour can dramatically affect cortical representations of the outside world, the extent to which any cortical coding might be considered purely sensory remains an important question. Moreover, it is possible that lowerdimensional signals are overly represented in visual areas, and that in other areas of the cortex (e.g. somatosensory for proprioception), the line between behaviour parameters and sensory processing is blurred.

    Many thanks for the clear and insightful summary of the results, significance and caveats of our analysis. We totally agree with this critical evaluation - and suggestions for future work.

  4. eLife

    Evaluation Summary:

    This work builds on rapidly accumulating evidence for the importance of measuring and accounting for behavior in neural data, and will be of interest to a broad neuroscience audience. Analyses of Allen Brain Atlas datasets show that sensory representations change and match up reliably with behavioral state. The manuscript's main conclusions are supported by the data and analyses and the work raises important questions about previous accounts of the sources of representational drift in sensory areas of the brain.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)

  5. eLife

    Reviewer #1 (Public Review):

    Sadeh and Clopath analyze two mouse datasets from the Allen Brain Atlas and show that sensory representations can have apparent representational drift that is entirely due to behavioral modulation. The analysis serves as a caution against over-interpreting shifts in the neural code. The analysis of data is coupled with careful modeling work that shows that the behavioral state reliably shifts sensory representations independently of stimulus modulation (rather than acting as a gain factor), and further show that it is reproducibly shifted when the behavioral state is adequately controlled for. The methods presented point towards a more careful consideration and measurement of behavioral states during sensory recordings, and a re-analysis of previous findings. The findings held up for both standard drifting grating stimuli as well as natural movies.

    The fact that neurons may have different tuning depending on the behavioral state of the animal raises obvious questions about readout. The authors show that neurons with strong behavioral shifts should simply be ignored and that this can be achieved if the downstream decoder weights inputs with more stimulus information. While questions remain about why behavior shifts representations and how that could be more effectively utilized by downstream circuits, the results presented clearly show that sensory representations might not always be simply drifting over time, and will spark some careful analysis of past and future experimental results.

  6. eLife

    Reviewer #2 (Public Review):

    Studies from recent years have shown that neuronal responses to the same stimuli or behavior can gradually change with time - a phenomenon known as representational drift. Other recent studies have shown that changes in behavior can also modulate neuronal responses to a given sensory stimulus. In this manuscript, Sadeh and Clopath analyzed publicly available data from the Allen Institute to examine the relationship between animal behavioral variability and changes in neuronal representations. The paper is timely and certainly has the potential to be of interest to neuroscientists working in different fields. However, there are currently several important issues with the analysis of the data and their interpretations that the authors should address. We believe that after these concerns are addressed, this study will be an important contribution to the field.

    1. The manuscript raises a potential problem: while previous work suggested that the passage of time leads to gradual changes in neuronal responses, the causality structure is different: i.e., the passage of time leads to gradual changes in behavior, which in turn lead to gradual changes in neuronal responses. The authors conclude that "variable behavioral signal might be misinterpreted as representational drift". While this may be true, in its current form, the paper lacks critical analyses that would support such a claim. It is possible that both factors - time and behavior - have a unique contribution to changes in neuronal responses, or that only time elicits changes in neuronal responses (and behavior is just correlated with time). Thus, the authors should demonstrate that these changes cannot be explained solely by the passage of time and elucidate the unique contributions of behavior (and elapsed time) to changes in representations.

    2. There are also several issues with the analysis of the data and the presentation of the results. The most concerning of which is that the data shows a non-linear (and non-monotonic) relationship between behavioral changes and representational similarity. In many of the presented cases, the data points fall into two or more discrete clusters. This can lead to the false impression that there is a monotonic relationship between the two variables, even though there is no (or even opposite) relationship within each cluster. This is a crucial point since the clusters of data points most likely represent different blocks that were separated in time (or separation between within-block and across-block comparisons).

    3. The authors also suggest that using measures of coding stability such as 'population-vector correlations' may be problematic for quantifying representational drift because it could be influenced by changes in the neuronal activity rates, which may be unrelated to the stimulus. We agree that it is important to carefully dissociate between the effects of behavior on changes in neuronal activity that are stimulus-dependent or independent, but we feel that the criticism raised by the authors ignores the findings of multiple previous papers, which (1) did not purely attribute the observed changes to the sensory component, and (2) did dissociate between stimulus-dependent changes (in the cells' tuning) and off-context/stimulus-independent changes (in the cells' activity rates).

    4. Another important issue relates to the interchangeable use of the terms 'representational drift' and 'representational similarity'. Representational similarity is a measure to identify changes in representations, and drift is one such change. This may confuse the reader and lead to the misconception that all changes in neuronal responses are representational drift.

  7. eLife

    Reviewer #3 (Public Review):

    Although it is increasingly realized that cortical neural representations are inherently unstable, the meaning of such "drift" can be difficult or impossible to interpret without knowing how the representations are being read out and used by the nervous system (i.e. how it contributes to what the experimental animal is actually doing now or in the future). Previous studies of representational drift have either ignored or explicitly rejected the contribution of what the animal is doing, mostly due to a lack of high-dimensional behavioural data. Here the authors use perhaps the most extensive open-source and rigorous neural data available to take a more detailed look at how behaviour affects cortical neural representations as they change over repeated presentations of the same visual stimuli.

    The authors apply a variety of analyses to the same two datasets, all of which convincingly point to behavioural measures having a large impact on changing neural representations. They also pit models against each other to address how behavioural and stimulus signals combine to influence representations, whether independently or through behaviour influencing the gain of stimuli. One analysis uses subsets of neurons to decode the stimulus, and the independent model correctly predicts the subset to use for better decoding. However, one caveat may be that the nervous system does not need to decode the stimulus from the cortex independently of behaviour; if necessary, this could be done elsewhere in the nervous system with a parallel stream of visual information.

    Overall the authors' claims are well-supported and this study should lead to a re-assessment of the concept of "representational drift". Nonetheless, a weakness of all analyses presented here is that they are all based on data in head-fixed mice that were passively viewing visual stimuli, such that it is unclear what relevance the behaviour has. Furthermore, the behavioural measurements available in the open-source dataset (pupil movements and running speed) are still a very low dimensional representation of what the mice were actually doing (e.g. detailed kinematics of all body movements and autonomic outputs). Thus, although the authors here as well as other large-scale neural recording studies in the past decade or so make it clear that relatively basic measures of behaviour can dramatically affect cortical representations of the outside world, the extent to which any cortical coding might be considered purely sensory remains an important question. Moreover, it is possible that lower-dimensional signals are overly represented in visual areas, and that in other areas of the cortex (e.g. somatosensory for proprioception), the line between behaviour parameters and sensory processing is blurred.

  8. Version published to 10.1101/2022.01.02.474731v1 on bioRxiv