Measuring Sequences of Representations with Temporally Delayed Linear Modelling

  1. Yunzhe Liu
  2. Raymond J Dolan
  3. Hector Luis Penagos-Vargas
  4. Zeb Kurth-Nelson
  5. Timothy Behrens

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

There are rich structures in off-task neural activity. For example, task related neural codes are thought to be reactivated in a systematic way during rest. This reactivation is hypothesised to reflect a fundamental computation that supports a variety of cognitive functions. Here, we introduce an analysis toolkit (TDLM) for analysing this activity. TDLM combines nonlinear classification and linear temporal modelling to testing for statistical regularities in sequences of neural representations. It is developed using non-invasive neuroimaging data and is designed to take care of confounds and maximize sequence detection ability. The method can be extended to rodent electrophysiological recordings. We outline how TDLM can successfully reveal human replay during rest, based upon non-invasive magnetoencephalography (MEG) measurements, with strong parallels to rodent hippocampal replay. TDLM can therefore advance our understanding of sequential computation and promote a richer convergence between animal and human neuroscience research.

Article activity feed

  1. eLife

    ###Reviewer #3:

    The methods used by the authors seem like potentially really useful tools for research on neural activity related to sequences of stimuli. We were excited to see that a new toolbox might be available for these sorts of problems, which are widespread. The authors touch on a number of interesting scenarios and raise relevant issues related to cross-validation and inference of statistical significance. However, given (1) the paucity of code that they've posted, and its specificity to specific exact data and (2) the large literature on latent variable models combined with surrogate data for significance testing, I would hesitate to call TDLM a "framework". Moreover, in trying to present it in this generic way, the authors have made it more difficult to understand exactly what they are doing.

    Overall: This paper presents a novel approach for detecting sequential patterns in neural data however it needs more context. What's the contribution overall? How and why is this analysis technique better than say Bayesian template matching? Why is it so difficult to understand the details of the method?

    Major Concerns:

    The first and most important problem with this paper is that it is intended (it appears) to be a more detailed and enhanced retelling of the author's 2019 Cell paper. If this is the case, then it's important that it also be clearer and easier to read and understand than that one was. The authors should follow the normal tradition in computational papers:

    Present a clear and thorough explanation of one use of the method (i.e., MEG observations with discrete stimuli), then present the next approach (i.e., sequences?) with all the details necessary to understand it.

    The authors should start each section with a mathematical explanation of the X's - the equation(s) that describes how they are derived from specific data. Much of the discussion of cross validation actually refers to this mapping.

    Equation 5 also needs a clearer explanation - it would be better to write it as a sum of matrices (because that is clearer) than with the strange "vec" notation. And TAUTO, TF and TR should be described properly - TAUTO is "the identity matrix", TF and TR are "shift matrices, with ones on the first upper and lower off diagonals".

    The cross validation schemes need a clear description. Preferably using something like a LaTeX "algorithm" box so that they are precisely explained.

    Recognizing the need to balance readability for a general reader and interest, perhaps the details could be given for the first few problems, and then for subsequent results, the detail could go into a Methods section. Alternatively, the methods section could be done away with (though some things, such as the MEG data acquisition methods are reasonably in the methods).

    Usually, we think about latent variable model problems from a generative perspective. The approach taken in this paper seems to be similar to a Kalman filter with a multinomial observation (which would be equivalent to the logistic regression?), but it's unclear. Making the connection to the extensive literature on dynamical latent variable models would be helpful.

    Minor concerns:

    1. Many of the figures, and some of the text are from the 2019 Cell paper. The methods text is copied verbatim without citation.

    2. The TLDM model is presented without context or comparison to other computational approaches employed to identify sequences. Is it also used in the 2016 Kurth-Nelson paper? How does it compare, e.g., to Bayesian template matching (in the case of hippocampal data)?

    3. Cite literature from recent systems neuroscience using hidden Markov models and related discrete state space approaches on neural activity.

    4. How does this method deal with a long sequence for which the intra-sequences have variance in their delta t's? Or data where the observations have some temporal lag relative to each other?

    5. In the "sequences of sequences" section, the authors talk about combining states into meta states. But then the example they give, it appears they just use their vanilla approach. This whole section belongs in a different place than a "supplemental note". The data need proper attribution, an IACUC/ethics statement, etc.

    6. While code can be useful, it is not archival in the same way equations are. Supplementary Note 1 should be in the Methods, and needs to be rewritten in such a way that it explains the steps (i.e., in an algorithm box) rather than just using code. Moreover, when the data generated via this code is used in the text, this section in the methods can be mentioned/linked.

  2. eLife

    ###Reviewer #2:

    This paper addresses the important overall issue of how to detect and quantify sequential structure in neural activity. Such sequences have been studied in the rodent hippocampus for decades, but it has recently become possible to detect them in human MEG (and perhaps even fMRI) data, generating much current excitement and promise in bringing together these fields.

    In this paper, the authors examine and develop in more detail the method previously published in their groundbreaking MEG paper (Liu et al. 2019). The authors demonstrate that by aiming their method at the level of decoded neural data (rather than the sensor-level data) it can be applied to a wide range of data types and settings, such as rodent ephys data, stimulating cross-fertilization. This generality is a strength and distinguishes this work from the typically ad hoc (study-specific) methods that are the norm; this paper could be a first step towards a more domain-general sequence detection method. A further strength is that the general linear modeling framework lends itself well to regressing out potential confounds such as autocorrelations, as the authors show.

    However, our enthusiasm for the paper is limited by several overall issues:

    1. It seems a major claim is that the current method is somehow superior to other methods (e.g. from the abstract: "designed to take care of confounds" implying that other methods do not do this, and "maximize sequence detection ability" implying that other methods are less effective at detection). But there is very little actual comparison with other methods made to substantiate this claim, particularly for sequences of more than two states which have been extensively used in the rodent replay literature (see Tingley and Peyrache, Proc Royal Soc B 2020 for a recent review of the rodent methods; different shuffling procedures are applied to identify sequenceness, see e.g. Farooq et al. Neuron 2019 and Foster, Ann Rev Neurosci 2017). The authors should compare their method to some others in order to support these claims, or at a minimum discuss how their method relates to/improves upon the state of the art.

    2. The scope or generality of the proposed method should be made more explicit in a number of ways. First, it seems the major example is from MEG data with a small number of discrete states; how does the method handle continuous variables and larger state spaces? (The rodent ephys example could potentially address this but not enough detail was provided to understand what was done; see specific comments below.) Second, it appears this method describes sequenceness for a large chunk of data, but cannot tell whether an individual event (such as a hippocampal sharp wave-ripple and associated spiking) forms a sequence not. Third, there is some inconsistency in the terminology regarding scope: are the authors aiming to detect any kind of temporal structure in neural activity (first sentence of "Overview of TDLM" section) which would include oscillations, or only sequences? These are not fatal issues but should be clearly delineated.

    3. The inference part of the work is potentially very valuable because this is an area that has been well studied in GLM/multiple regression type problems. However, the authors limit themselves to asking "first-order" sequence questions (i.e. whether observed sequenceness is different from random) when key questions -- including whether or not there is evidence of replay -- are actually "second-order" questions because they require a comparison of sequenceness across two conditions (e.g. pre-task and post-task; I'm borrowing this terminology from van der Meer et al. Proc Royal Soc B 2020). The authors should address how to make this kind of comparison using their method.

    Minor Comments:

    1. Some discussion of grounding the question of what is considered a sequence should be included. What may look like a confound to a modeler may or may not be impacting downstream readout neurons; without access to a neural readout it is not a priori clear what our statistical methods "should" be detecting.

    2. The abstract emphasizes hippocampal replay, but no actual analysis of this is done. I don't think performing such analysis is necessary (although it would be a good way to compare this method to others) but the two should be more aligned.

    3. In the "Statistical Inference" section, the authors stated "Permuting time destroys the temporal smoothness of neural data, creating an artificially narrow null distribution...". Did the authors try shift shuffles, which shifts the time dimension of each row rather than randomly permuting it, hence breaking the relationship between variables but keeping their autocorrelation?

    4. In the "Regularization" section, it is hard to tell how L1 outperforms L2 in terms of detecting sequenceness without benchmarking them with ground truth. Are the authors doing this by quantifying decoding performance on withheld task data? Van der Meer et al. Hippocampus 2017 examine this issue for hippocampal place cell data.

    5. As a rodent ephys person I was excited about the application to hippocampal place cell data, but I couldn't understand Figure 5d and the associated supplementary description. In order for me to evaluate this component of the ms, substantially more explanation is needed on how the data is preprocessed and arranged, and what the analysis pipeline looks like. For instance, Is the left plot in Fig. 5d an average of all pairwise sequences (each decoded location with its neighbors)? And the right plot is the timescale at which this sequence repeats? If so, the repeat frequency should be at rat theta frequency or a little faster (because of phase precession) so I would expect 9 or 10 Hz max -- surprised to see what looks like 12 Hz? In the Supplementary note, I found the discussion about running direction distracting, wouldn't it be simpler and easier to understand to analyze only one direction to start? Also, please clarify if the sequence algorithm was run on the raw decoded probabilities, or on the maximum a posteriori (MAP) locations. What happens if there are no spikes in a given time bin (likely to happen with a small 10 ms window) and were putative interneurons excluded (they should be)? Finally, the authors should note that theta sequences can arise from independent spiking of phase precessing neurons (Chadwick et al. eLife 2015) which seems exactly the kind of issue that the multiple regression framework of TDLM should be able to elucidate; what covariates could be added into the model to test Chadwick et al's claim?

  3. eLife

    ###Reviewer #1:

    This paper describes temporal delayed linear modelling (TDLM), a method for detecting sequential replay during awake rest periods in human neuroimaging data. The method involves first training a classifier to decode states from labeled data, then building linear models that quantify the extent to which one state predicts the next expected state at particular lags, and finally assessing reliability by running the analysis with permuted labels.

    This method has already been fruitfully used in prior empirical papers by the authors, and this paper serves to present the details of the method and code such that others may make use of it. Based on existing findings, the method seems extremely promising, with potential for widespread interest and adoption in the human neuroimaging community. The paper would benefit, however, from more discussion of the scope of the applicability of the method and its relationship to methods already available in the rodent and (to a lesser extent) human literature.

    1. TDLM is presented as a general tool for detecting replay, with special utility for noninvasive human neuroimaging modalities. The method is tested mainly on MEG data, with one additional demonstration in rodent electrophysiology. Should researchers expect to be able to apply the method directly to EEG or fMRI data? If not, what considerations or modifications would be involved?

    2. How does the method relate to the state of the art methods for detecting replay in electrophysiology data? What precludes using those methods in MEG data or other noninvasive modalities? And conversely, do the authors believe animal replay researchers would benefit from adopting the proposed method?

    3. It would be useful for the authors to comment on the applicability of the method to sleep data, especially as rodent replay decoding methods are routinely used during both awake rest and sleep.

    4. How does the method relate to the Wittkuhn & Schuck fMRI replay detection method? What might be the advantages and disadvantages of each?

    5. The authors make the point that spatial correlation as well as anti-correlation between state patterns reduces the ability to detect sequences. The x axis for Fig 3c begins at zero, demonstrating that lower positive correlation is better than higher positive correlation. Given the common practice of building one classifier to decode multiple states (as opposed to a separate classifier for each state), it would be very useful to provide a demonstration that the relationship in Fig 3c flips (more correlation is better for sequenceness) when spatial correlations are in the negative range.

    6. In the Results, the authors specify using a single time point for spatial patterns, which would seem to be a potentially very noisy estimate. In the Methods, they explain that the data were downsampled from 600 to 100 Hz to improve SNR. It seems likely that downsampling or some other method of increasing SNR will be important for the use of single time point estimates. It would be useful for the authors to comment on this and provide recommendations in the Results section.

    7. While the demonstration that the method works for detecting theta sequences in navigating rodents is very useful, the paper is missing the more basic demonstration that it works for simple replay during awake rest in rodents. This would be important to include to the extent that the authors believe the method will be of use in comparing replay between species.

    8. The authors explain that they "had one condition where we measured resting activity before the subjects saw any stimuli. Therefore, by definition these stimuli could not replay, but we can use the classifiers from these stimuli (measured later) to test the false positive performance of statistical tests on replay." My understanding of the rodent preplay literature is that you might indeed expect meaningful "replay" prior to stimulus exposure, as existing sequential dynamics may be co-opted to represent subsequent stimulus sequences. It may therefore be tricky to assume no sequenceness prior to stimulus exposure.

  4. eLife

    ##Preprint Review

    This preprint was reviewed using eLife’s Preprint Review service, which provides public peer reviews of manuscripts posted on bioRxiv for the benefit of the authors, readers, potential readers, and others interested in our assessment of the work. This review applies only to version 3 of the manuscript.

    ###Summary: The reviewers all felt that the work is extremely valuable: a domain-general replay detection method would be of wide interest and utility. However, as it stands, the paper is lacking context and comparisons to existing methods. Most importantly, the paper would have a larger impact if comparisons with standard replay methods were included. The paper would also benefit from additional detail in the description of the methods and data.

  5. Version published to 10.1101/2020.04.30.066407 on bioRxiv