Constructing the hierarchy of predictive auditory sequences in the marmoset brain
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
This paper will be of interest to neuroscientists interested in predictive coding. By using complementary neuroimaging and electrophysiological methods to measure brain-wide activation patterns in marmosets in response to sound pattern violations, the authors provide evidence for the hierarchical organization of predictive coding across subcortical and cortical levels of the auditory pathway.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. The reviewers remained anonymous to the authors.)
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
Our brains constantly generate predictions of sensory input that are compared with actual inputs, propagate the prediction-errors through a hierarchy of brain regions, and subsequently update the internal predictions of the world. However, the essential feature of predictive coding, the notion of hierarchical depth and its neural mechanisms, remains largely unexplored. Here, we investigated the hierarchical depth of predictive auditory processing by combining functional magnetic resonance imaging (fMRI) and high-density whole-brain electrocorticography (ECoG) in marmoset monkeys during an auditory local-global paradigm in which the temporal regularities of the stimuli were designed at two hierarchical levels. The prediction-errors and prediction updates were examined as neural responses to auditory mismatches and omissions. Using fMRI, we identified a hierarchical gradient along the auditory pathway: midbrain and sensory regions represented local, shorter-time-scale predictive processing followed by associative auditory regions, whereas anterior temporal and prefrontal areas represented global, longer-time-scale sequence processing. The complementary ECoG recordings confirmed the activations at cortical surface areas and further differentiated the signals of prediction-error and update, which were transmitted via putative bottom-up γ and top-down β oscillations, respectively. Furthermore, omission responses caused by absence of input, reflecting solely the two levels of prediction signals that are unique to the hierarchical predictive coding framework, demonstrated the hierarchical top-down process of predictions in the auditory, temporal, and prefrontal areas. Thus, our findings support the hierarchical predictive coding framework, and outline how neural networks and spatiotemporal dynamics are used to represent and arrange a hierarchical structure of auditory sequences in the marmoset brain.
Article activity feed
-
-
Author Response:
Reviewer #1 (Public Review):
The paper clearly indicates that by using parallel fMRI and ECoG experiments, the authors are able to detail the hierarchy of predictive coding in the cortical and higher subcortical areas of the auditory pathway. The methodology is well detailed and I didn't spot any major concerns.
The scientific methodology detailed in this paper appears to be sound. Further, the main conclusions appear to be well argued.
We thank the reviewer for the positive comments.
The statistical analysis, however, is not reported clearly in the main text. For instance, I'm unsure how multiple comparison correction was addressed. A more detailed primer on the statistical methods used in the results section is warranted.
In the fMRI analysis, to assess the novelty responses as a function of the different …
Author Response:
Reviewer #1 (Public Review):
The paper clearly indicates that by using parallel fMRI and ECoG experiments, the authors are able to detail the hierarchy of predictive coding in the cortical and higher subcortical areas of the auditory pathway. The methodology is well detailed and I didn't spot any major concerns.
The scientific methodology detailed in this paper appears to be sound. Further, the main conclusions appear to be well argued.
We thank the reviewer for the positive comments.
The statistical analysis, however, is not reported clearly in the main text. For instance, I'm unsure how multiple comparison correction was addressed. A more detailed primer on the statistical methods used in the results section is warranted.
In the fMRI analysis, to assess the novelty responses as a function of the different number of sequence types, we performed a within-subject one-way ANOVA design in SPM, where the single-session contrast images corresponding to trial types were introduced as within-subject factor. To directly compare the responses of different novelties, we defined each type of contrast using the pair-wise t-test. We initially observed the results with a threshold of uncorrected p < 0.001 at the voxel level, and then considered the results as significant at p < 0.05 with false discovery rate (FDR) corrected for multiple comparisons across the brain (Cacciaglia et al., 2019; Uhrig et al., 2014). If no voxel survived FDR correction, then a threshold of uncorrected p < 0.001 was used. In the ECoG analysis, we performed the independent-sample t-test for each comparison in TFRs. The multiple comparison problem originates from the fact that ECoG data are multidimensional. For ECoG data from a single electrode, the signals are sampled at multiple frequencies and multiple time points. Therefore, we used a nonparametric cluster-based permutation test for multiple comparisons over frequency and time (Chao et al., 2018; El Karoui et al., 2015). To report the statistical analysis more clearly, we have added the details about statistical methods of fMRI and ECoG analyses, and multiple comparison correction, in the results of the main text. Please see the section of 1 st -level (local) novelty (xY sequences).
My largest concerns are to do with communication, and language overreach. At one point the term "lower auditory pathways" is used, but the lowest portion investigated in this study is the IC, and this usage was in reference to the thalamus. There's a lot of brain between the IC and cochlea, to say nothing of the thalamus. There are also concerns about both the temporal and spatial resolution of fMRI and ECoG - the text at times implies that the resolution for these techniques is far greater than it is. However, these are communication issues that should be easily addressed.
We are grateful for the reviewer’s suggestions. The temporal and spatial terms we used in the last version were based on our paradigm and recording methods. With 9.4 T fMRI, the lowest area of the auditory pathway that we can assess is the midbrain. Thus, we limited the observed range of the auditory pathway from midbrain to frontal cortex (as shown in Figure 1C). In the design of our paradigm, the local auditory information focus on the millisecond timescale, and the global auditory information refer to the second timescale. Therefore, we defined the temporal range from millisecond to second. However, we have realized that the temporal and spatial terms we used were not rigorous and may cause ambiguity. Throughout the revised manuscript, we have rewritten them as lower- and higher-level areas, shorter- and longer-time scales.
Reviewer #2 (Public Review):
In this study, Jiang et al. combined whole-brain 9.4 T functional magnetic resonance imaging and large-scale electrocorticography to study brain wide activation patterns in response to different pattern violations in marmosets. The authors confirm previous results of a cortical hierarchy for auditory predictive processing and expand on these results by quantifying subcortical responses in MGB and IC as well as using omission to confirm previous results obtained with mismatches. The results highlight the existence of the two levels of auditory prediction signals in the marmoset brain that can be interpreted in a hierarchical predictive processing framework.
The paradigm used to assess the hierarchical depth of predictive auditory sequences for processing predictions errors and prediction updates at two distinct timescales is well designed, and presumably based on one of the authors earlier studies (Chao et al., 2018). Unfortunately, the current study fails to highlight the novelty of this work (as far as we can tell, mainly the omission responses) and give adequate credit to previous work on the topic. However, this can be easily fixed by rephrasing the relevant passages of the manuscript.
We thank the reviewer for the positive comments. We have now revised the Results and Discussion to provide more details about the omission responses and discuss the contribution and novelty of omission sequences in the hierarchical predictive coding. Please also see the reply to Q1 of the Main concerns.
Main concerns:
- It would be good to clarify what the novelty of the present manuscript is (omission responses) in comparison to the previous work (Chao et al., (2018)). The authors do argue that their higher resolution fMRI, allows them to also study subcortical response - which is correct - but the authors make no use of them in any meaningful way in the manuscript. The emphasis on novelty is likely better placed on the omission responses.
We thank the reviewer for the constructive suggestion. In the revised Discussion, in comparison to the previous work (Chao et al., 2018), we have specifically emphasized the novelties of the present study.
-First, the model described the 1st - and 2nd - levels of violations (prediction and error) in the present study is novel and more straightforward. Instead of using the partial- or full-global predictions in the Chao et al. model, which is challenging to interpret, we first introduce the sequences with xx and xY as separate internal templates. Similarly, as we mentioned in the discussion, although the local-global paradigm has been intensively studied in humans and macaque monkeys (Chao et al., 2018; Uhrig et al., 2014; Wacongne et al., 2011), most studies tested the global violation by combining xx|xY and xY|xx novelties, which, in fact, contain two different types of predictions. Our study is the first to separate the two novelties and search for their neural representations, respectively. This is important because xx|xY novelty was only involved in the 2nd-level signal with the xY sequence as the internal sequence template, and the xY|xx novelty was involved in both 1st - and 2nd -level signals (the 1st-level novelty triggers the 2nd-level novelty), where the xx sequence was the internal template (see Discussion).
-Second, this is the first study to construct the hierarchy of predictive auditory sequences in the marmoset brain using fMRI. Our results extended the hierarchical organization of predictive coding from the cortex to the subcortical regions. To emphasize the importance of this animal model, we have added a section of Marmosets as an animal model for auditory sequences in the Discussion.
-Third, most importantly, as suggested by the referee, the omission responses is indeed novel. To highlight it, we have expanded the results of omission and provided more discussion of its contribution to the hierarchical predictive coding.
- Figure 3C (and all similar figures). We fear this figure is not interpretable without a substantially improved explanation. Both what the arrows mean (i.e. how they are computed), and what the values indicate that are listed next to the arrows is not explained (arrows appear randomly bi- or unidirectional and the legend at the bottom of the figure is not very helpful).
We apologize for the missing details in Figure 3C. The color dots in the brain diagrams indicate the electrodes with significant responses found in corresponding comparisons, which were subsequently used in the functional correlation test (see Materials and Methods). Lines represent significant functional correlations between signals from the paired brain diagrams. Labeled values close to lines provide the Pearson correlation coefficient (p-value) of the corresponding correlations. Unidirectional arrows indicate relative temporal orders at which the signals appear, while bidirectional arrows indicate uncertain temporal orders of the signals. Figure 3D, 5C and D, Figure 3-figure supplement 1C and D, Figure 5-figure supplement 1C and D have the same format as Figure 3C. Accordingly, we have revised the legend of Figures 3C and D, 5C and D and added more explanations in the Results.
References:
Cacciaglia, R., Costa-Faidella, J., Zarnowiec, K., Grimm, S., & Escera, C. (2019, Feb 1). Auditory predictions shape the neural responses to stimulus repetition and sensory change. Neuroimage, 186, 200-210. https://doi.org/10.1016/j.neuroimage.2018.11.007
Chao, Z. C., Takaura, K., Wang, L., Fujii, N., & Dehaene, S. (2018, Dec 5). Large-Scale Cortical Networks for Hierarchical Prediction and Prediction Error in the Primate Brain. Neuron, 100(5), 1252-1266.e1253. https://doi.org/10.1016/j.neuron.2018.10.004
El Karoui, I., King, J. R., Sitt, J., Meyniel, F., Van Gaal, S., Hasboun, D., Adam, C., Navarro, V., Baulac, M., Dehaene, S., Cohen, L., & Naccache, L. (2015, Nov). Event-Related Potential, Timefrequency, and Functional Connectivity Facets of Local and Global Auditory Novelty Processing: An Intracranial Study in Humans. Cereb Cortex, 25(11), 4203-4212. https://doi.org/10.1093/cercor/bhu143
Uhrig, L., Dehaene, S., & Jarraya, B. (2014, Jan 22). A hierarchy of responses to auditory regularities in the macaque brain. J Neurosci, 34(4), 1127-1132. https://doi.org/10.1523/jneurosci.3165- 13.2014
Wacongne, C., Labyt, E., van Wassenhove, V., Bekinschtein, T., Naccache, L., & Dehaene, S. (2011, Dec 20). Evidence for a hierarchy of predictions and prediction errors in human cortex. Proc Natl Acad Sci U S A, 108(51), 20754-20759. https://doi.org/10.1073/pnas.1117807108
-
Evaluation Summary:
This paper will be of interest to neuroscientists interested in predictive coding. By using complementary neuroimaging and electrophysiological methods to measure brain-wide activation patterns in marmosets in response to sound pattern violations, the authors provide evidence for the hierarchical organization of predictive coding across subcortical and cortical levels of the auditory pathway.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. The reviewers remained anonymous to the authors.)
-
Reviewer #1 (Public Review):
The paper clearly indicates that by using parallel fMRI and ECoG experiments, the authors are able to detail the hierarchy of predictive coding in the cortical and higher subcortical areas of the auditory pathway. The methodology is well detailed and I didn't spot any major concerns.
The scientific methodology detailed in this paper appears to be sound. Further, the main conclusions appear to be well argued.
The statistical analysis, however, is not reported clearly in the main text. For instance, I'm unsure how multiple comparison correction was addressed. A more detailed primer on the statistical methods used in the results section is warranted.
My largest concerns are to do with communication, and language overreach. At one point the term "lower auditory pathways" is used, but the lowest portion …
Reviewer #1 (Public Review):
The paper clearly indicates that by using parallel fMRI and ECoG experiments, the authors are able to detail the hierarchy of predictive coding in the cortical and higher subcortical areas of the auditory pathway. The methodology is well detailed and I didn't spot any major concerns.
The scientific methodology detailed in this paper appears to be sound. Further, the main conclusions appear to be well argued.
The statistical analysis, however, is not reported clearly in the main text. For instance, I'm unsure how multiple comparison correction was addressed. A more detailed primer on the statistical methods used in the results section is warranted.
My largest concerns are to do with communication, and language overreach. At one point the term "lower auditory pathways" is used, but the lowest portion investigated in this study is the IC, and this usage was in reference to the thalamus. There's a lot of brain between the IC and cochlea, to say nothing of the thalamus. There are also concerns about both the temporal and spatial resolution of fMRI and ECoG - the text at times implies that the resolution for these techniques is far greater than it is. However, these are communication issues that should be easily addressed.
-
Reviewer #2 (Public Review):
In this study, Jiang et al. combined whole-brain 9.4 T functional magnetic resonance imaging and large-scale electrocorticography to study brain wide activation patterns in response to different pattern violations in marmosets. The authors confirm previous results of a cortical hierarchy for auditory predictive processing and expand on these results by quantifying subcortical responses in MGB and IC as well as using omission to confirm previous results obtained with mismatches. The results highlight the existence of the two levels of auditory prediction signals in the marmoset brain that can be interpreted in a hierarchical predictive processing framework.
The paradigm used to assess the hierarchical depth of predictive auditory sequences for processing predictions errors and prediction updates at two …
Reviewer #2 (Public Review):
In this study, Jiang et al. combined whole-brain 9.4 T functional magnetic resonance imaging and large-scale electrocorticography to study brain wide activation patterns in response to different pattern violations in marmosets. The authors confirm previous results of a cortical hierarchy for auditory predictive processing and expand on these results by quantifying subcortical responses in MGB and IC as well as using omission to confirm previous results obtained with mismatches. The results highlight the existence of the two levels of auditory prediction signals in the marmoset brain that can be interpreted in a hierarchical predictive processing framework.
The paradigm used to assess the hierarchical depth of predictive auditory sequences for processing predictions errors and prediction updates at two distinct timescales is well designed, and presumably based on one of the authors earlier studies (Chao et al., 2018). Unfortunately, the current study fails to highlight the novelty of this work (as far as we can tell, mainly the omission responses) and give adequate credit to previous work on the topic. However, this can be easily fixed by rephrasing the relevant passages of the manuscript.
Main concerns:
It would be good to clarify what the novelty of the present manuscript is (omission responses) in comparison to the previous work (Chao et al., (2018)). The authors do argue that their higher resolution fMRI, allows them to also study subcortical response - which is correct - but the authors make no use of them in any meaningful way in the manuscript. The emphasis on novelty is likely better placed on the omission responses.
Figure 3C (and all similar figures). We fear this figure is not interpretable without a substantially improved explanation. Both what the arrows mean (i.e. how they are computed), and what the values indicate that are listed next to the arrows is not explained (arrows appear randomly bi- or unidirectional and the legend at the bottom of the figure is not very helpful).
-
