Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream
Curation statements for this article:-
Curated by eLife
eLife assessment
This study presents a useful application of a prior model published by the authors to a new dataset. The results from this approach were interesting and solid but the conclusions that one can make from the application of the model to only one paper are limited in scope and would depend on further probing to know if the model itself has face validity as a model of ventral visual stream function.
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (eLife)
Abstract
Decades of neuroscientific research has sought to understand medial temporal lobe (MTL) involvement in perception. Apparent inconsistencies in the literature have led to competing interpretations of the available evidence; critically, findings from human participants with naturally occurring MTL damage appear to be inconsistent with data from monkeys with surgical lesions. Here, we leverage a ‘stimulus-computable’ proxy for the primate ventral visual stream (VVS), which enables us to formally evaluate perceptual demands across stimulus sets, experiments, and species. With this modeling framework, we analyze a series of experiments administered to monkeys with surgical, bilateral damage to perirhinal cortex (PRC), an MTL structure implicated in visual object perception. Across experiments, PRC-lesioned subjects showed no impairment on perceptual tasks; this originally led us(Eldridge et al., 2018) to conclude that PRC is not involved in perception. Here, we find that a ‘VVS-like’ model predicts both PRC-intact and -lesioned choice behaviors, suggesting that a linear readout of the VVS should be sufficient for performance on these tasks. Evaluating these computational results alongside findings from human experiments, we suggest that results from (Eldridge et al., 2018) alone cannot be used as evidence against PRC involvement in perception. These data indicate that experimental findings from human and non-human primates are consistent. As such, what appeared to be discrepancies between species was in fact due to reliance on informal accounts of perceptual processing.
Article activity feed
-
-
Author Respone
Reviewer #1 (Public Review):
This article describes the application of a computational model, previously published in 2021 in Neuron, to an empirical dataset from monkeys, previously published in 2018 in eLife. The 2021 modeling paper argued that the model can be used to determine whether a particular task depends on the perirhinal cortex as opposed to being soluble using ventral visual stream structures alone. The 2018 empirical paper used a series of visual discrimination tasks in monkeys that were designed to contain high levels of 'feature ambiguity' (in which the stimuli that must be discriminated share a large proportion of overlapping features), and yet animals with rhinal cortex lesions were unimpaired, leading the authors to conclude that perirhinal cortex is not involved in the visual perception of objects. …
Author Respone
Reviewer #1 (Public Review):
This article describes the application of a computational model, previously published in 2021 in Neuron, to an empirical dataset from monkeys, previously published in 2018 in eLife. The 2021 modeling paper argued that the model can be used to determine whether a particular task depends on the perirhinal cortex as opposed to being soluble using ventral visual stream structures alone. The 2018 empirical paper used a series of visual discrimination tasks in monkeys that were designed to contain high levels of 'feature ambiguity' (in which the stimuli that must be discriminated share a large proportion of overlapping features), and yet animals with rhinal cortex lesions were unimpaired, leading the authors to conclude that perirhinal cortex is not involved in the visual perception of objects. The present article revisits and revises that conclusion: when the 2018 tasks are run through the 2021 computational model, the model suggests that they should not depend on perirhinal cortex function after all, because the model of VVS function achieves the same levels of performance as both controls and PRC-lesioned animals from the 2018 paper. This leads the authors of the present study to conclude that the 2018 data are simply "non-diagnostic" in terms of the involvement of the perirhinal cortex in object perception.
We appreciate the Reviewer’s careful reading and synthesis of the background and general findings of this manuscript.
The authors have successfully applied the computational tool from 2021 to empirical data, in exactly the way the tool was designed to be used. To the extent that the model can be accepted as a veridical proxy for primate VVS function, its conclusions can be trusted and this study provides a useful piece of information in the interpretation of often contradictory literature. However, I found the contribution to be rather modest. The results of this computational study pertain to only a single empirical study from the literature on perirhinal function (Eldridge et al, 2018). Thus, it cannot be argued that by reinterpreting this study, the current contribution resolves all controversy or even most of the controversy in the foregoing literature. The Bonnen et al. 2021 paper provided a potentially useful computational tool for evaluating the empirical literature, but using that tool to evaluate (and ultimately rule out as non-diagnostic) a single study does not seem to warrant an entire manuscript: I would expect to see a reevaluation of a much larger sample of data in order to make a significant contribution to the literature, above and beyond the paper already published in 2021. In addition, the manuscript in its current form leaves the motivations for some analyses under-specified and the methods occasionally obscure.
We believe that our comments outline our rationale for focusing our current analysis on data from Eldridge et al. In brief, these data provide compelling evidence against PRC involvement in perception, and are the only such data with PRC-lesioned/-intact macaques that we were able to secure the stimuli for. As such, data from Eldridge et al. provide a singular opportunity to address discrepancies between human and macaque lesion data. For this reason, we propose the current work as a Research Advance Article type, building off of a manuscript that was previously published in eLife.
Reviewer #2 (Public Review):
The goal of this paper is to use a model-based approach, developed by one of the authors and colleagues in 2021, to critically re-evaluate the claims made in a prior paper from 2018, written by the other author of this paper (and colleagues), concerning the role of perirhinal cortex in visual perception. The prior paper compared monkeys with and without lesions to the perirhinal cortex and found that their performance was indistinguishable on a difficult perceptual task (categorizing dog-cat morphs as dogs or cats). Because the performance was the same, the conclusion was that the perirhinal cortex is not needed for this task, and probably not needed for perception in general, since this task was chosen specifically to be a task that the perirhinal cortex might be important for. Well, the current work argues that in fact the task and stimuli were poorly chosen since the task can be accomplished by a model of the ventral visual cortex. More generally, the authors start with the logic that the perirhinal cortex gets input from the ventral visual processing stream and that if a task can be performed by the ventral visual processing stream alone, then the perirhinal cortex will add no benefit to that task. Hence to determine whether the perirhinal cortex plays a role in perception, one needs a task (and stimulus set) that cannot be done by the ventral visual cortex alone (or cannot be done at the level of monkeys or humans).
There are two important questions the authors then address. First, can their model of the ventral visual cortex perform as well as macaques (with no lesion) on this task? The answer is yes, based on the analysis of this paper. The second question is, are there any tasks that humans or monkeys can perform better than their ventral visual model? If not, then maybe the ventral visual model (and biological ventral visual processing stream) is sufficient for all recognition. The answer here too is yes, there are some tasks humans can perform better than the model. These then would be good tasks to test with a lesion approach to the perirhinal cortex. It is worth noting, though, that none of the analyses showing that humans can outperform the ventral visual model are included in this paper - the papers which showed this are cited but not discussed in detail.
Major strength:
The computational and conceptual frameworks are very valuable. The authors make a compelling case that when patients (or animals) with perirhinal lesions perform equally to those without lesions, the interpretation is ambiguous: it could be that the perirhinal cortex doesn't matter for perception in general, or it could be that it doesn't matter for this stimulus set. They now have a way to distinguish these two possibilities, at least insofar as one trusts their ventral visual model (a standard convolutional neural network). While of course, the model cannot be perfectly accurate, it is nonetheless helpful to have a concrete tool to make a first-pass reasonable guess at how to disambiguate results. Here, the authors offer a potential way forward by trying to identify the kinds of stimuli that will vs won't rely on processing beyond the ventral visual stream. The re-interpretation of the 2018 paper is pretty compelling.
We thank the Reviewer for the careful reading of our manuscript and for providing a fantistics synthesis of the current work.
Major weakness:
It is not clear that an off-the-shelf convolution neural network really is a great model of the ventral visual stream. Among other things, it lacks eccentricity-dependent scaling. It also lacks recurrence (as far as I could tell).
We agree with the Reviewer completely on this point: there is little reason to expect that off-the-shelf convolutional neural networks should predict neural responses from the ventral visual stream, for the reasons outlined above (no eccentricity-dependent scaling, no recurrence) as well as others (weight sharing is biologically implausible, as well as the data distributions and objective functions use to optimize these models). Perhaps surprisingly, these models do provide quantitatively accurate accounts of information processing throughout the VVS; while this is well established within the literature, we were careless to simply assert this as a given without providing an account of these data. We appreciate the Reviewer for making this clear and we have changed the manuscript in several critical ways in order to avoid making unsubstantiated claims in the current version. We hope that these changes also make it easier for the casual reader to appreciate the logic in our analyses. First, in the introduction, we outline some of the prior experimental work that demonstrates how deep learning models are effective proxies for neural responses throughout the VVS. We also demonstrate this model-neural fit in the current paper using electrophysiological recordings, but also including comments about the limitation of these models raised by the Reviewer.
In the introduction we also more clearly demarcate prior contributions from our recent computational work, and highlight how models approximate the performance supported by a linear readout of the VVS, but fail to reach human-level performance.
Results from these analyses were essential to understanding the logic of the paper but previously (as noted by the Reviewer) this critical evidence was cited but not directly presented. We include a description to these we describe these data in the introduction more thoroughly, and substantial change Figure 1, in order to visualize these data (b).
Moreover, we include a over of the methods and data used to generate these plots in the results and methods sections.
While there is little reason to expect that off-the-shelf convolutional neural networks should predict neural responses from the ventral visual stream, we believe that these modifications to the manuscript (to the introduction and figure one, as well as the results and methods sections) make clear that these models are nonetheless useful methods for predicting VVS responses and the behaviors that depend on the VVS.
To the authors' credit, they show detailed analysis on an image-by-image basis showing that in fine detail the model is not a good approximation of monkey choice behavior. This imposes limits on how much trust one should put in model performance as a predictor of whether the ventral visual cortex is sufficient to do a task or not. For example, suppose the authors had found that their model did more poorly than the monkeys (lesioned or not lesioned). According to their own logic, they would have, it seems, been led to the interpretation that some area outside of the ventral visual cortex (but not the perirhinal cortex) contributes to perception, when in fact it could have simply been that their model missed important aspects of ventral visual processing. That didn't happen in this paper, but it is a possible limitation of the method if one wanted to generalize it. There is work suggesting that recurrence in neural networks is essential for capturing the pattern of human behavior on some difficult perceptual judgments (e.g., Kietzmann et al 2019, PNAS). In other words, if the ventral model does not match human (or macaque) performance on some recognition task, it does not imply that an area outside the ventral stream is needed - it could just be that a better ventral model (eg with recurrence, or some other property not included in the model) is needed. This weakness pertains to the generalizability of the approach, not to the specific claims made in this paper, which appear sound.
We could not agree more with the Reviewer on these points. It could have been the case that these models' lack of correspondence with known biological properties (e.g. recurrence) led them to lack something important about VVS-supported performance, and that this would derail the entire modeling effort here. Surprisingly, this has not been the case, as is evident in the clear correspondence between model performance and monkey data in Eldridge et al. 2018. Nonetheless, we would expect that other experimental paradigms should be able to reveal these model failings. And future work evaluating PRC involvement in perception must contend with this very problem in order to move forward with this modeling framework. That is, it is of critical importance that these VVS models and the VVS itself exhibit similar failure modes, otherwise it is not possible to use these models to isolate behaviors that may depend on PRC.
A second issue is that the title of the paper, "Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream" does not seem to be supported by the paper. The paper challenges a conclusion about macaque lesion data. What inconsistency is reconciled, and how?
It appears that this point was lost in the original manuscript; we have tried to clarify this idea in both the abstract and the introduction. In summary, the cumulative evidence from the human lesion data suggest that PRC is involved in visual object perception, while there are still studies in the monkey literature that suggest otherwise (e.g. Eldridge et al. 2018). In this manuscript, we suggest that this apparent inconsistency is, in fact, simply a consequence of reliance on information interpretations of the monkey lesion data.
We have made substantive changes to the abstract so this is an obvious, central claim.
We have also made substantive changes to the introduction to make resolving this cross-species discrepancy a more central aim of the current manuscript.
-
eLife assessment
This study presents a useful application of a prior model published by the authors to a new dataset. The results from this approach were interesting and solid but the conclusions that one can make from the application of the model to only one paper are limited in scope and would depend on further probing to know if the model itself has face validity as a model of ventral visual stream function.
-
Reviewer #1 (Public Review):
This article describes the application of a computational model, previously published in 2021 in Neuron, to an empirical dataset from monkeys, previously published in 2018 in eLife. The 2021 modeling paper argued that the model can be used to determine whether a particular task depends on the perirhinal cortex as opposed to being soluble using ventral visual stream structures alone. The 2018 empirical paper used a series of visual discrimination tasks in monkeys that were designed to contain high levels of 'feature ambiguity' (in which the stimuli that must be discriminated share a large proportion of overlapping features), and yet animals with rhinal cortex lesions were unimpaired, leading the authors to conclude that perirhinal cortex is not involved in the visual perception of objects. The present article …
Reviewer #1 (Public Review):
This article describes the application of a computational model, previously published in 2021 in Neuron, to an empirical dataset from monkeys, previously published in 2018 in eLife. The 2021 modeling paper argued that the model can be used to determine whether a particular task depends on the perirhinal cortex as opposed to being soluble using ventral visual stream structures alone. The 2018 empirical paper used a series of visual discrimination tasks in monkeys that were designed to contain high levels of 'feature ambiguity' (in which the stimuli that must be discriminated share a large proportion of overlapping features), and yet animals with rhinal cortex lesions were unimpaired, leading the authors to conclude that perirhinal cortex is not involved in the visual perception of objects. The present article revisits and revises that conclusion: when the 2018 tasks are run through the 2021 computational model, the model suggests that they should not depend on perirhinal cortex function after all, because the model of VVS function achieves the same levels of performance as both controls and PRC-lesioned animals from the 2018 paper. This leads the authors of the present study to conclude that the 2018 data are simply "non-diagnostic" in terms of the involvement of the perirhinal cortex in object perception.
The authors have successfully applied the computational tool from 2021 to empirical data, in exactly the way the tool was designed to be used. To the extent that the model can be accepted as a veridical proxy for primate VVS function, its conclusions can be trusted and this study provides a useful piece of information in the interpretation of often contradictory literature. However, I found the contribution to be rather modest. The results of this computational study pertain to only a single empirical study from the literature on perirhinal function (Eldridge et al, 2018). Thus, it cannot be argued that by reinterpreting this study, the current contribution resolves all controversy or even most of the controversy in the foregoing literature. The Bonnen et al. 2021 paper provided a potentially useful computational tool for evaluating the empirical literature, but using that tool to evaluate (and ultimately rule out as non-diagnostic) a single study does not seem to warrant an entire manuscript: I would expect to see a reevaluation of a much larger sample of data in order to make a significant contribution to the literature, above and beyond the paper already published in 2021. In addition, the manuscript in its current form leaves the motivations for some analyses under-specified and the methods occasionally obscure.
-
Reviewer #2 (Public Review):
The goal of this paper is to use a model-based approach, developed by one of the authors and colleagues in 2021, to critically re-evaluate the claims made in a prior paper from 2018, written by the other author of this paper (and colleagues), concerning the role of perirhinal cortex in visual perception. The prior paper compared monkeys with and without lesions to the perirhinal cortex and found that their performance was indistinguishable on a difficult perceptual task (categorizing dog-cat morphs as dogs or cats). Because the performance was the same, the conclusion was that the perirhinal cortex is not needed for this task, and probably not needed for perception in general, since this task was chosen specifically to be a task that the perirhinal cortex *might* be important for. Well, the current work …
Reviewer #2 (Public Review):
The goal of this paper is to use a model-based approach, developed by one of the authors and colleagues in 2021, to critically re-evaluate the claims made in a prior paper from 2018, written by the other author of this paper (and colleagues), concerning the role of perirhinal cortex in visual perception. The prior paper compared monkeys with and without lesions to the perirhinal cortex and found that their performance was indistinguishable on a difficult perceptual task (categorizing dog-cat morphs as dogs or cats). Because the performance was the same, the conclusion was that the perirhinal cortex is not needed for this task, and probably not needed for perception in general, since this task was chosen specifically to be a task that the perirhinal cortex *might* be important for. Well, the current work argues that in fact the task and stimuli were poorly chosen since the task can be accomplished by a model of the ventral visual cortex. More generally, the authors start with the logic that the perirhinal cortex gets input from the ventral visual processing stream and that if a task can be performed by the ventral visual processing stream alone, then the perirhinal cortex will add no benefit to that task. Hence to determine whether the perirhinal cortex plays a role in perception, one needs a task (and stimulus set) that cannot be done by the ventral visual cortex alone (or cannot be done at the level of monkeys or humans).
There are two important questions the authors then address. First, can their model of the ventral visual cortex perform as well as macaques (with no lesion) on this task? The answer is yes, based on the analysis of this paper. The second question is, are there any tasks that humans or monkeys can perform better than their ventral visual model? If not, then maybe the ventral visual model (and biological ventral visual processing stream) is sufficient for all recognition. The answer here too is yes, there are some tasks humans can perform better than the model. These then would be good tasks to test with a lesion approach to the perirhinal cortex. It is worth noting, though, that none of the analyses showing that humans can outperform the ventral visual model are included in this paper - the papers which showed this are cited but not discussed in detail.
Major strength:
The computational and conceptual frameworks are very valuable. The authors make a compelling case that when patients (or animals) with perirhinal lesions perform equally to those without lesions, the interpretation is ambiguous: it could be that the perirhinal cortex doesn't matter for perception in general, or it could be that it doesn't matter for this stimulus set. They now have a way to distinguish these two possibilities, at least insofar as one trusts their ventral visual model (a standard convolutional neural network). While of course, the model cannot be perfectly accurate, it is nonetheless helpful to have a concrete tool to make a first-pass reasonable guess at how to disambiguate results. Here, the authors offer a potential way forward by trying to identify the kinds of stimuli that will vs won't rely on processing beyond the ventral visual stream. The re-interpretation of the 2018 paper is pretty compelling.Major weakness:
It is not clear that an off-the-shelf convolution neural network really is a great model of the ventral visual stream. Among other things, it lacks eccentricity-dependent scaling. It also lacks recurrence (as far as I could tell). To the authors' credit, they show detailed analysis on an image-by-image basis showing that in fine detail the model is not a good approximation of monkey choice behavior. This imposes limits on how much trust one should put in model performance as a predictor of whether the ventral visual cortex is sufficient to do a task or not. For example, suppose the authors had found that their model did more poorly than the monkeys (lesioned or not lesioned). According to their own logic, they would have, it seems, been led to the interpretation that some area outside of the ventral visual cortex (but not the perirhinal cortex) contributes to perception, when in fact it could have simply been that their model missed important aspects of ventral visual processing. That didn't happen in this paper, but it is a possible limitation of the method if one wanted to generalize it. There is work suggesting that recurrence in neural networks is essential for capturing the pattern of human behavior on some difficult perceptual judgments (e.g., Kietzmann et al 2019, PNAS). In other words, if the ventral model does not match human (or macaque) performance on some recognition task, it does not imply that an area outside the ventral stream is needed - it could just be that a better ventral model (eg with recurrence, or some other property not included in the model) is needed. This weakness pertains to the generalizability of the approach, not to the specific claims made in this paper, which appear sound.A second issue is that the title of the paper, "Inconsistencies between human and macaque lesion data can be resolved with a stimulus-computable model of the ventral visual stream" does not seem to be supported by the paper. The paper challenges a conclusion about macaque lesion data. What inconsistency is reconciled, and how?
-
