Brain representations of motion and position in the double-drift illusion

  1. Noah J Steinberg
  2. Zvi N Roth
  3. J Anthony Movshon
  4. Elisha Merriam

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This study is an elegant imaging experiment in humans that shows that visual area hMT+, but not other candidate brain areas, signal the perceived motion path in a visual drift illusion. Using a powerful computational decoding approach, the results show a perceptual representation of the illusory position in space for moving stimuli even when the actual retinal position of the stimulus is kept stable. Such a representation and the underlying neural mechanisms are of broad importance for our understanding of the neural basis of sensory perception.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their names with the authors.)

Reviewed by

Read the full article

Listed in

Log in to save this article

Abstract

In the ‘double-drift’ illusion, local motion within a window moving in the periphery of the visual field alters the window’s perceived path. The illusion is strong even when the eyes track a target whose motion matches the window so that the stimulus remains stable on the retina. This implies that the illusion involves the integration of retinal signals with non-retinal eye-movement signals. To identify where in the brain this integration occurs, we measured BOLD fMRI responses in visual cortex while subjects experienced the double-drift illusion. We then used a combination of univariate and multivariate decoding analyses to identify (1) which brain areas were sensitive to the illusion and (2) whether these brain areas contained information about the illusory stimulus trajectory. We identified a number of cortical areas that responded more strongly during the illusion than a control condition that was matched for low-level stimulus properties. Only in area hMT+ was it possible to decode the illusory trajectory. We additionally performed a number of important controls that rule out possible low-level confounds. Concurrent eye tracking confirmed that subjects accurately tracked the moving target; we were unable to decode the illusion trajectory using eye position measurements recorded during fMRI scanning, ruling out explanations based on differences in oculomotor behavior. Our results provide evidence for a perceptual representation in human visual cortex that incorporates extraretinal information.

Article activity feed

  1. Version published to 10.7554/elife.76803 on eLife
  2. eLife

    Author Response

    Reviewer #1 (Public Review):

    Combining functional MRI with a decoder, the authors probe the neural substrate of the double drift illusion in visual cortex. Their elegant behavioural paradigm keeps the actual retinal position of the stimulus stable while inducing the illusion with a combination of smooth pursuit and visual motion. The results show that the illusory drift path can be decoded from a signal in extrastriate visual area hMT+ but not other visual areas. Importantly, this can be done in the absence of spatial attention to the stimulus location.

    The particular strengths of this study lie in the elegant paradigm and the clear attentional control. The methodology of the decoder is powerful and at the same time straightforward, well explained, and well accepted in the literature. A potential weakness of the study is the lack of simultaneous eye movement recordings in the scanner. Such data could have provided further clarification of the potential underlying neural mechanism and whether differences in eye movements could contribute to the decoding of the visual illusion path. There are some controls that mitigate this.

    We have addressed the Reviewer's comment by repeating the fMRI experiment in a new group of subjects in which we were able to also obtain concurrent, high-quality eye tracking. When we initially conducted the experiment, it was not possible to perform eye tracking in the 7T scanner at NIH. Because of this limitation, we were forced to depend on careful eye tracking in a pre-scan behavioral experiment. But in the ensuing period of time, we have developed a protocol for obtaining high quality eye tracking with an Eyelink 1000 mounted in the bore of the scanner. Now that we have the ability to collect concurrent eye tracking, we repeated the fMRI experiment and found that our main fMRI result replicated (i.e, it was possible to decode the direction of the illusion from fMRI responses in hMT+). Additional, the concurrent fMRI eye tracking enabled us to make four important observations (see new Fig 4):

    First, subjects maintained stable fixation when the target was stationary during fixation and accurately pursued the vertically moving target during illusion (Fig 4). This analysis confirms that the drifting Gabor remained at a relatively fixed position on the retina during the illusory period.

    Second, there were no differences in microsaccades between any of the conditions. We quantified the direction, amplitude, and frequency of all saccades for each condition. While we did observe small rightward microsaccades, none of the microsaccade characteristics differed between conditions. The rightward microsaccades may have been due to the sustained eccentric leftward fixation. Or, it may have been due to attention to the right visual field stimulus (despite the foveal attention task). Or it may have reflected the known horizontal microsaccade bias. Regardless, we do not believe our fMRI results are related to microsaccades because these small saccades did not differ across condition.

    Finally, we wondered if small not-easily-quantified ocular deviations could have differed between conditions, and somehow result in differences in fMRI activity picked up by the decoding analysis. To test for this possibility, we trained a classier to discriminate condition based on the raw eye traces (just as we did in the main fMRI data analysis). But unlike the fMRI analysis, we found that it was not possible to decode the direction of the illusion from the eye traces themselves.

    We conclude that the ability to decode the illusion from fMRI responses were not due to differences in eye movements caused by the illusion.

    The authors provide important evidence for a potential neural substrate in the extrastriate visual cortex for encoding the perceived spatial location of a moving stimulus. This significantly extends previous studies that showed relevant spatiotopic signals outside visual cortex. Understanding the neural substrate and the underlying neural mechanisms for encoding perceived spatiotopic location are of broad importance for our understanding of the neural basis of sensory perception.

    We thank the Editor for this positive assessment of our work.

    Reviewer #3 (Public Review):

    The authors studied the neural basis of the double drift illusion, an illusion in which a Gabor drifting both horizontally within an aperture and moving vertically along a path appears to follow a diagonal trajectory, perceptually displaced off its true vertical path in the direction of the horizontal drift. The illusion is strong and its neural basis is intriguing. The authors suggest it can be used to address the locus of spatiotopic processing in the brain. They find that fMRI BOLD activity in hMT+ can be used to decode the illusory drift direction of the stimulus, even under conditions of withdrawn attention. They internally replicate this result and ensure it is not due to local motion. They interpret the finding to indicate that hMT+ contains spatiotopic information. This was a carefully designed and conducted study, and the manuscript writing and figures are clear.

    Despite the care that went into the study design and control experiments, I see some potential interpretational issues, and I am uncertain about the scientific advance. My main questions are about the interpretation of the findings, the possible confound of smooth pursuit eye movements, and the relation to previous studies, including previous fMRI studies of the same illusion. I also would like to see more thorough reporting of behavior.

    Major comments

    1. The authors motivate the study by saying that there have been conflicting results about which brain areas are involved in spatiotopic coding, but they did not give an indication about why there might be conflicting results or why the current study is suitable to address the previous discrepancies. Is this study simply adding another observation to the existing body of literature, or does it go beyond previous studies in a critical theoretical way?

    There have indeed been conflicting results in the literature. One idea that has received some prior support in the literature is that spatiotopic location information can depend on the task. Our experiment tests this idea by measuring cortical responses during an illusion that involves spatiotopic coding. Previous human fMRI studies reporting spatiotopic coding have not really linked cortical activity with the perception of spatiotopic coordinates. Hence, we feel that our results make a unique contribution to the field.

    1. The authors interpret the finding of illusory drift direction encoding in hMT+ to mean that hMT+ is coding the illusory spatial position of the stimulus. But could an alternative explanation be that hMT+ is coding the illusory global motion direction, and not the spatial position per se? If this is a possible account, then the result would still indicate that an illusory motion percept is reflected in hMT+ but it would seem not to answer the question about spatiotopic coding which motivated the paper.

    Here, the Reviewer suggests an interesting alternative explanation—that responses in MT pertain to the direction of global motion rather than stimulus position. However, this alternative possibility would still involve spatiotopic coding. In order for the brain to compute the direction of global motion of a stimulus that is at a fixed retinal position, some spatiotopic computation must occur. So, we do not agree with the Reviewers suggestion that this alternative explanation undermines the motivation of this study.

    1. It is good that the authors sought to rule out the possibility that smooth pursuit eye movements were driving the decoding results in hMT+, but I'm not sure they have yet convincingly done so. Decoding based on the pursuit selective voxels alone was very nearly significant (p = 0.052), which was not acknowledged in the text of the paper. Furthermore, because voxels that were both pursuit and stimulus selective were excluded from the pursuit selective ROI, decoding performance in that ROI may have been underestimated.

    To clarify, voxels that were identified by both localizers were NOT excluded from either ROI. When we repeated decoding (from Expt 2, Fig 3B) using disjoint voxel selection (i.e., analyzing voxels that only responded in the stim localizer, or only responded in the pursuit localizer, and excluding voxels that responded to both), we obtained qualitatively similar results, although the magnitude of the effects were smaller, which is not surprising given the much smaller number of voxels remaining in the ROI, and hence the disjoint ROIs only proved marginally significant in MT for the stim localizer (p=0.049).

    1. A previous fMRI study of the double drift illusion (Liu et al. 2019 Current Biology) also found above chance decoding of illusory drift direction in hMT+. The authors mention this study but do not discuss it, so it was unclear to me what the advance is of the current study over that study. The main differences I see are that in the current study, 1) the observer is also moving their eyes so that the double drift stimulus is theoretically stabilized on the retina, and 2) attention is withdrawn from the stimulus. But in both studies, hMT+ contains information about the illusory drift direction even though retinotopic information is the same, so it's not clear to me that the differences between these studies lead to fundamentally different interpretations.

    The results of Liu et al. are not relevant to the reference frame used to encode the stimulus. Because subjects were fixating in Liu et al., the encoding of the illusion could have been in either retinal or spatiotopic coordinates. In our study, the stimulus must have been encoded in spatiotopic coordinates. One interesting feature of Liu et al. is the issue of cross decoding the illusion and actual percept (training the decoder on veridical motion of different angles, and then testing the decoder on data collected during the illusion). One potentially interesting extension of the cross decoding approach would be to train the decoder on a version of the illusion involving fixation (as in Liu et al), but then testing the decoder on the illusion during pursuit. One would expect cross decoding if spatiotopic coordinates are used in both cases. We now discuss this possibility (Discussion: Relationship to a previous study of the double-drift illusion).

  3. eLife

    Evaluation Summary:

    This study is an elegant imaging experiment in humans that shows that visual area hMT+, but not other candidate brain areas, signal the perceived motion path in a visual drift illusion. Using a powerful computational decoding approach, the results show a perceptual representation of the illusory position in space for moving stimuli even when the actual retinal position of the stimulus is kept stable. Such a representation and the underlying neural mechanisms are of broad importance for our understanding of the neural basis of sensory perception.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 and Reviewer #2 agreed to share their names with the authors.)

  4. eLife

    Reviewer #1 (Public Review):

    Combining functional MRI with a decoder, the authors probe the neural substrate of the double drift illusion in visual cortex. Their elegant behavioural paradigm keeps the actual retinal position of the stimulus stable while inducing the illusion with a combination of smooth pursuit and visual motion. The results show that the illusory drift path can be decoded from a signal in extrastriate visual area hMT+ but not other visual areas. Importantly, this can be done in the absence of spatial attention to the stimulus location.

    The particular strengths of this study lie in the elegant paradigm and the clear attentional control. The methodology of the decoder is powerful and at the same time straightforward, well explained, and well accepted in the literature. A potential weakness of the study is the lack of simultaneous eye movement recordings in the scanner. Such data could have provided further clarification of the potential underlying neural mechanism and whether differences in eye movements could contribute to the decoding of the visual illusion path. There are some controls that mitigate this.

    The authors provide important evidence for a potential neural substrate in the extrastriate visual cortex for encoding the perceived spatial location of a moving stimulus. This significantly extends previous studies that showed relevant spatiotopic signals outside visual cortex. Understanding the neural substrate and the underlying neural mechanisms for encoding perceived spatiotopic location are of broad importance for our understanding of the neural basis of sensory perception.

  5. eLife

    Reviewer #2 (Public Review):

    This is a very cute experiment. The authors take advantage of the "double drift illusion", which works even when tracking, implying that non-retinal signals can drive the illusion; they then show that hMT+ can decode the direction of the illusory, non-retinal motion. They perform several important controls, confirming that hMT+ responds to non-retinal motion signals rather than retinal artifacts. Having said that the manuscript in its present form needs a major rewrite and reorganization. It has clearly been written for a short note, which does not do justice to the large literature on the subject.

  6. eLife

    Reviewer #3 (Public Review):

    The authors studied the neural basis of the double drift illusion, an illusion in which a Gabor drifting both horizontally within an aperture and moving vertically along a path appears to follow a diagonal trajectory, perceptually displaced off its true vertical path in the direction of the horizontal drift. The illusion is strong and its neural basis is intriguing. The authors suggest it can be used to address the locus of spatiotopic processing in the brain. They find that fMRI BOLD activity in hMT+ can be used to decode the illusory drift direction of the stimulus, even under conditions of withdrawn attention. They internally replicate this result and ensure it is not due to local motion. They interpret the finding to indicate that hMT+ contains spatiotopic information. This was a carefully designed and conducted study, and the manuscript writing and figures are clear.

    Despite the care that went into the study design and control experiments, I see some potential interpretational issues, and I am uncertain about the scientific advance. My main questions are about the interpretation of the findings, the possible confound of smooth pursuit eye movements, and the relation to previous studies, including previous fMRI studies of the same illusion. I also would like to see more thorough reporting of behavior.

    Major comments

    1. The authors motivate the study by saying that there have been conflicting results about which brain areas are involved in spatiotopic coding, but they did not give an indication about why there might be conflicting results or why the current study is suitable to address the previous discrepancies. Is this study simply adding another observation to the existing body of literature, or does it go beyond previous studies in a critical theoretical way?

    2. The authors interpret the finding of illusory drift direction encoding in hMT+ to mean that hMT+ is coding the illusory spatial position of the stimulus. But could an alternative explanation be that hMT+ is coding the illusory global motion direction, and not the spatial position per se? If this is a possible account, then the result would still indicate that an illusory motion percept is reflected in hMT+ but it would seem not to answer the question about spatiotopic coding which motivated the paper.

    3. It is good that the authors sought to rule out the possibility that smooth pursuit eye movements were driving the decoding results in hMT+, but I'm not sure they have yet convincingly done so. Decoding based on the pursuit selective voxels alone was very nearly significant (p = 0.052), which was not acknowledged in the text of the paper. Furthermore, because voxels that were both pursuit and stimulus selective were excluded from the pursuit selective ROI, decoding performance in that ROI may have been underestimated.

    4. A previous fMRI study of the double drift illusion (Liu et al. 2019 Current Biology) also found above chance decoding of illusory drift direction in hMT+. The authors mention this study but do not discuss it, so it was unclear to me what the advance is of the current study over that study. The main differences I see are that in the current study, 1) the observer is also moving their eyes so that the double drift stimulus is theoretically stabilized on the retina, and 2) attention is withdrawn from the stimulus. But in both studies, hMT+ contains information about the illusory drift direction even though retinotopic information is the same, so it's not clear to me that the differences between these studies lead to fundamentally different interpretations.

  7. Version published to 10.1101/2022.01.25.477714v1 on bioRxiv