Activity in perirhinal and entorhinal cortex predicts perceived visual similarities among category exemplars with highest precision
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
This study examines the degree to which idiosyncrasies in visual object representations (beyond just image-driven objective representations) exist in visual areas of the brain. The authors report that later stages of the visual processing stream (specifically involving the perirhinal cortex and parts of the entorhinal cortex) do show these idiosyncratic representations, and for all levels of similarity (even for distinguishing very highly similar stimuli). These findings are interesting to vision scientists working to understand the role of different regions within the visual processing stream and to memory scientists interested in how this visual input is transformed in medial temporal lobe regions.
This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
- Neuroscience (eLife)
Abstract
Vision neuroscience has made great strides in understanding the hierarchical organization of object representations along the ventral visual stream (VVS). How VVS representations capture fine-grained visual similarities between objects that observers subjectively perceive has received limited examination so far. In the current study, we addressed this question by focussing on perceived visual similarities among subordinate exemplars of real-world categories. We hypothesized that these perceived similarities are reflected with highest fidelity in neural activity patterns downstream from inferotemporal regions, namely in perirhinal (PrC) and anterolateral entorhinal cortex (alErC) in the medial temporal lobe. To address this issue with functional magnetic resonance imaging (fMRI), we administered a modified 1-back task that required discrimination between category exemplars as well as categorization. Further, we obtained observer-specific ratings of perceived visual similarities, which predicted behavioural discrimination performance during scanning. As anticipated, we found that activity patterns in PrC and alErC predicted the structure of perceived visual similarity relationships among category exemplars, including its observer-specific component, with higher precision than any other VVS region. Our findings provide new evidence that subjective aspects of object perception that rely on fine-grained visual differentiation are reflected with highest fidelity in the medial temporal lobe.
Article activity feed
-
-
-
Author Response:
Reviewer #1 (Public Review):
This manuscript begins with the larger notion that comparing item similarity is an important principle that guides human behavior, and that these similarity representations can have both a general and an idiosyncratic component. While individual-specific representations have been identified in some visual processing areas using personally meaningful object stimuli or simple stimulus features, this study looks at complex real-world stimuli with no personal meaning. They observe that even using the same stimuli across people, there are differences in how people judge the similarity of same-category items, and these differences correlate with performance on comparing the identities of images presented in sequence. Further, they examine the visual stream – specifically early visual areas like …
Author Response:
Reviewer #1 (Public Review):
This manuscript begins with the larger notion that comparing item similarity is an important principle that guides human behavior, and that these similarity representations can have both a general and an idiosyncratic component. While individual-specific representations have been identified in some visual processing areas using personally meaningful object stimuli or simple stimulus features, this study looks at complex real-world stimuli with no personal meaning. They observe that even using the same stimuli across people, there are differences in how people judge the similarity of same-category items, and these differences correlate with performance on comparing the identities of images presented in sequence. Further, they examine the visual stream – specifically early visual areas like the early visual cortex (EVC) and lateral occipital complex (LOC), and late visual areas like the perirhinal cortex (PRC) and the anterior lateral entorhinal cortex (alErC), to see how representations in the brain relate to these behavioral representations. They observe that while EVC and LOC show correlations with behavior, PRC and alErC really show the strongest links to the individual-specific representation and to fine-grained differences across stimuli.
The analyses in this paper are very methodologically sound. They rely on well-controlled and well-tested analyses (e.g., testing if representational similarity is indeed higher for comparisons with one own's behavior, versus someone else's). They also replicate their results using classification-based analyses. My only key methodological question is more about experimental design. Given that participants performed the object arrangement task right before entering the scanner, I wonder if similarities in the brain to their own behavior could be due to memory for the representation they created just prior (especially given the role of PRC and alErC in memory). So, if instead, participants were shown and interacted with someone else's similarity arrangement, I wonder if these regions would show more similarity to that other person's arrangement, or still show similarity with one's own representations. It is thus currently unclear if the current findings are due to some deep-seated individual, internal representations, or memory for a recently performed task.
We thank the reviewer for highlighting numerous strengths in our methodological approach. We note that our experimental design was intentionally designed to have participants complete the iMDS sorting task prior to completion of the 1-Back task in the scanner. This ensured that all participants had the same exemplar familiarity during scanning. We cannot rule out the possibility that this order led to priming effects, as suggested by the Reviewer, which may have facilitated the emergence of observer-specific effects. Notably, however, such priming effects could be expected to affect all exemplars equally whose neural representations were probed during scanning. Critically, we also obtained new behavioural evidence for this resubmission, now included in Supplementary Materials, revealing that reports of perceptual similarity in our iMDS task reflect an observer characteristic that is temporally stable rather than just a situational idiosyncrasy. In the follow-up behavioural experiment (Supplementary Figure 2; page 49), “a distinct group of 30 participants completed two sessions of the iMDS task for the 10 object categories separated by 7 days +/- 1 day later. Correlations were computed between each participants’ perceived similarity Representational Dissimilarity Matrices (RDMs) from Session 1 and from Session 2. The mean within-subject correlation across the two sessions was 0.84, indicating high stability of participant’s perceived similarity ratings one week apart. Intersubject correlations for perceived similarity ratings across all exemplars and categories. Correlations were computed between each participant’s RDM Session 1 with the mean RDM (excluding the participant) in Session 2. Mean inter-subject correlation was 0.68. Critically, a paired t-test(intra-subject>inter-subject correlation) confirmed that intra-subject correlations were significantly higher than inter-subject correlations (p<0.0001). This pattern of results indicates that the perceived similarity structure that is unique to the individual observer is a stable characteristic.” This new behavioural experiment provides critical support for the perspective that our findings elucidate individual differences equivalent to those discussed as observer-specific effects in the vision literature more broadly (Mollon, Boston, Peterzell, & Webster, 2017, Individual differences in visual science: What can be learned and what is good experimental practice? Vision Research).
The results presented in this work are very clear and fit in well with previous findings on idiosyncracies in visual areas (Charest et al., 2014), and various work on the PRC as it relates to oddball tasks and object representational similarity. One question I am stuck with in this work is whether these current results show us something surprising or new. I'm unsure if we would have expected anything different for these generic real-world stimuli (versus the personally meaningful stimuli, or limited visual features tested previously).
We acknowledge that the paper by Charest et al., 2014 was an important stepping stone for the work presented in our manuscript. We have modified the framing of our main research questions so as to place more emphasis on levels (or grain) of perceived similarity among category exemplars that are reflected in subjective reports and object representations in different VVS regions. We also place more emphasis on the representational-hierarchical model as the theoretical framework that allows for related predictions and that guides our interpretation. In the interpretation of our results in the Discussion, we also speculate that observer-specific effects in fine-grained similarity perception, and in corresponding representations in PrC and alErC, may reflect interindividual differences in category expertise. Here we make reference to recent behavioural findings (i.e., Collins & Behrmann, 2020; Minos, Ferko, & Kohler, 2021) and present hypotheses about neural representations that can be directly tested in future fMRI studies with training paradigms. Indeed, we are in the process of planning to conduct such follow-up work in our laboratory. Inasmuch as the current study (i) revealed a mapping of perceived similarities among exemplars of object categories to representations in PrC and alErC (regions traditionally not considered to be part of the VVS and not included in analyses reported by Charest and colleagues); (ii) was guided by the representational-hierarchical model of VVS organization for interpretation of findings in PrC/alErC vs more posterior regions; (iii) showed an impact of observer-specific perceived similarities on behaviour that was most pronounced for fine-grained discrimination; and (iv) involved computational modeling to help interpret differences between observer-specific and observer-general (i.e., averaged) representations in different VVS regions, we feel that the contribution of our study clearly goes beyond replicating idiosyncrasies in VVS object representations as previously reported in Charest and colleagues’ pioneering work.
The manuscript frames the study around the idea that similarities are an important guiding principle of behavior. But this statement is not necessarily so obvious to me – is judging similarity itself an important ecological behavior, or is it just that looking at similarity structures can give insight into underlying relationships in how we represent information? (The latter is how I often see these sorts of representational similarity analyses.) What is this similarity task really capturing about our representations for these objects, and why do these idiosyncrasies emerge? My main hesitation about the current work is that I struggle with seeing a scope beyond a replication (e.g., finding behavior-correlated idiosyncracies in the brain, but with a different stimulus set, and in a slightly similar but expected region). I really want to know what factors are driving these idiosyncracies (e.g., is it visual? mnemonic? semantic?), and what this implies about the mechanisms of the PRC and ERC.
This is a really interesting point to consider in the context of prior lesion research on the role of PrC in perceptual discrimination. In this rich literature (see Murray et al., 2007 for review) emphasis has been placed on the perception of similarities between objects as probed with oddity-discrimination tasks, which require observers (human or nonhuman) to judge perceived similarities among multiple objects. Findings of behavioural impairments on this task after medial temporal-lobe lesions that included PrC and ErC have played a key role in the development of the representational-hierarchical model that guides the interpretation of our research. While the results of this lesion research have critically informed theoretical arguments that PrC plays a role in perceptual discrimination of objects (see Bonnen et al., Neuron, 2021, for a recent computationally focused review), it is important to recognize that they do not provide a characterization of similarity structure of neural representations in PrC and alErC, nor a characterization of the transformation of representations from more posterior VVS regions to these regions in the medial temporal lobe. Moreover, lesion findings do not address whether neural representations in the medial temporal lobe capture the perceived similarity structure that is unique to individual observers. Critically, our behavioural results also directly reveal effects of the perceived similarities that observers reported on discrimination performance in the 1-back task we employed during scanning. Inasmuch as this task taps discrimination between exemplars of real-world categories we would argue that the examination of representational similarity structure in our study also sheds lights on ecologically relevant behaviour.
-
Evaluation Summary:
This study examines the degree to which idiosyncrasies in visual object representations (beyond just image-driven objective representations) exist in visual areas of the brain. The authors report that later stages of the visual processing stream (specifically involving the perirhinal cortex and parts of the entorhinal cortex) do show these idiosyncratic representations, and for all levels of similarity (even for distinguishing very highly similar stimuli). These findings are interesting to vision scientists working to understand the role of different regions within the visual processing stream and to memory scientists interested in how this visual input is transformed in medial temporal lobe regions.
This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also …
Evaluation Summary:
This study examines the degree to which idiosyncrasies in visual object representations (beyond just image-driven objective representations) exist in visual areas of the brain. The authors report that later stages of the visual processing stream (specifically involving the perirhinal cortex and parts of the entorhinal cortex) do show these idiosyncratic representations, and for all levels of similarity (even for distinguishing very highly similar stimuli). These findings are interesting to vision scientists working to understand the role of different regions within the visual processing stream and to memory scientists interested in how this visual input is transformed in medial temporal lobe regions.
This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.
-
Reviewer #1 (Public Review):
This manuscript begins with the larger notion that comparing item similarity is an important principle that guides human behavior, and that these similarity representations can have both a general and an idiosyncratic component. While individual-specific representations have been identified in some visual processing areas using personally meaningful object stimuli or simple stimulus features, this study looks at complex real-world stimuli with no personal meaning. They observe that even using the same stimuli across people, there are differences in how people judge the similarity of same-category items, and these differences correlate with performance on comparing the identities of images presented in sequence. Further, they examine the visual stream – specifically early visual areas like the early visual …
Reviewer #1 (Public Review):
This manuscript begins with the larger notion that comparing item similarity is an important principle that guides human behavior, and that these similarity representations can have both a general and an idiosyncratic component. While individual-specific representations have been identified in some visual processing areas using personally meaningful object stimuli or simple stimulus features, this study looks at complex real-world stimuli with no personal meaning. They observe that even using the same stimuli across people, there are differences in how people judge the similarity of same-category items, and these differences correlate with performance on comparing the identities of images presented in sequence. Further, they examine the visual stream – specifically early visual areas like the early visual cortex (EVC) and lateral occipital complex (LOC), and late visual areas like the perirhinal cortex (PRC) and the anterior lateral entorhinal cortex (alErC), to see how representations in the brain relate to these behavioral representations. They observe that while EVC and LOC show correlations with behavior, PRC and alErC really show the strongest links to the individual-specific representation and to fine-grained differences across stimuli.
The analyses in this paper are very methodologically sound. They rely on well-controlled and well-tested analyses (e.g., testing if representational similarity is indeed higher for comparisons with one own's behavior, versus someone else's). They also replicate their results using classification-based analyses. My only key methodological question is more about experimental design. Given that participants performed the object arrangement task right before entering the scanner, I wonder if similarities in the brain to their own behavior could be due to memory for the representation they created just prior (especially given the role of PRC and alErC in memory). So, if instead, participants were shown and interacted with someone else's similarity arrangement, I wonder if these regions would show more similarity to that other person's arrangement, or still show similarity with one's own representations. It is thus currently unclear if the current findings are due to some deep-seated individual, internal representations, or memory for a recently performed task.
The results presented in this work are very clear and fit in well with previous findings on idiosyncracies in visual areas (Charest et al., 2014), and various work on the PRC as it relates to oddball tasks and object representational similarity. One question I am stuck with in this work is whether these current results show us something surprising or new. I'm unsure if we would have expected anything different for these generic real-world stimuli (versus the personally meaningful stimuli, or limited visual features tested previously). The manuscript frames the study around the idea that similarities are an important guiding principle of behavior. But this statement is not necessarily so obvious to me – is judging similarity itself an important ecological behavior, or is it just that looking at similarity structures can give insight into underlying relationships in how we represent information? (The latter is how I often see these sorts of representational similarity analyses.) What is this similarity task really capturing about our representations for these objects, and why do these idiosyncrasies emerge? My main hesitation about the current work is that I struggle with seeing a scope beyond a replication (e.g., finding behavior-correlated idiosyncracies in the brain, but with a different stimulus set, and in a slightly similar but expected region). I really want to know what factors are driving these idiosyncracies (e.g., is it visual? mnemonic? semantic?), and what this implies about the mechanisms of the PRC and ERC.
-
Reviewer #2 (Public Review):
The current manuscript describes a novel application of RSA to an fMRI dataset comprising task-related responses to real-world object stimuli. The authors scanned younger adults (n=23) while they discriminated between 4 different object exemplars for 10 different categories; participants also completed subjective similarity ratings between all pairs of objects, based on a validated iMDS approach (Kriegeskorte & Mur, 2013). The value of this work rests on 2 main findings. First, perirhinal and entorhinal cortices predicted subjective similarity, even when item comparisons were restricted to exemplar comparisons of different levels of similarity. Second, the subjective similarities predicted discrimination performance in an observer-specific manner, such that Brain-Model similarities are higher within- than …
Reviewer #2 (Public Review):
The current manuscript describes a novel application of RSA to an fMRI dataset comprising task-related responses to real-world object stimuli. The authors scanned younger adults (n=23) while they discriminated between 4 different object exemplars for 10 different categories; participants also completed subjective similarity ratings between all pairs of objects, based on a validated iMDS approach (Kriegeskorte & Mur, 2013). The value of this work rests on 2 main findings. First, perirhinal and entorhinal cortices predicted subjective similarity, even when item comparisons were restricted to exemplar comparisons of different levels of similarity. Second, the subjective similarities predicted discrimination performance in an observer-specific manner, such that Brain-Model similarities are higher within- than across-individuals-and only in PRC and ERC. There are many follow-up and supplementary analyses (whole-brain searchlight analysis, SNR calculations), all of which help to support the main findings.
-