Task-related hemodynamic responses in human early visual cortex are modulated by task difficulty and behavioral performance

  1. Charlie S Burlingham
  2. Minyoung Ryoo
  3. Zvi N Roth
  4. Saghar Mirbagheri
  5. David J Heeger
  6. Elisha P Merriam

  • Curated by eLife

    eLife logo

    Evaluation Summary:

    This work will be of general interest to those using hemodynamic imaging, such as fMRI, to study the brain. A hemodynamic signature that is modulated by arousal level changes on a trial-to-trial basis, such as those evoked by a difficult task, would both provide insight into arousal influences on cortical activity and characterize a prominent signal in hemodynamic data that is rarely considered. Overall, the data and analyses provide support for this idea, but would benefit from additional analysis, controls, and a better framework for integrating this work with the existing literature and mechanistic understanding of arousal, neural activity, and behavior.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)

Reviewed by

Read the full article See related articles

Listed in

Log in to save this article

Abstract

Early visual cortex exhibits widespread hemodynamic responses in the absence of visual stimulation, which are entrained to the timing of a task and not predicted by local spiking or local field potential. Such task-related responses (TRRs) covary with reward magnitude and physiological signatures of arousal. It is unknown, however, if TRRs change on a trial-to-trial basis according to behavioral performance and task difficulty. If so, this would suggest that TRRs reflect arousal on a trial-to-trial timescale and covary with critical task and behavioral variables. We measured functional magnetic resonance imaging blood-oxygen-level-dependent (fMRI-BOLD) responses in the early visual cortex of human observers performing an orientation discrimination task consisting of separate easy and hard runs of trials. Stimuli were presented in a small portion of one hemifield, but the fMRI response was measured in the ipsilateral hemisphere, far from the stimulus representation and focus of spatial attention. TRRs scaled in amplitude with task difficulty, behavioral accuracy, reaction time, and lapses across trials. These modulations were not explained by the influence of respiration, cardiac activity, or head movement on the fMRI signal. Similar modulations with task difficulty and behavior were observed in pupil size. These results suggest that TRRs reflect arousal and behavior on the timescale of individual trials.

Article activity feed

  1. Version published to 10.7554/elife.73018 on eLife
  2. eLife

    Author Response:

    Reviewer #1:

    While the implications are compelling, a few other controls and analyses would better establish the link between arousal level and the TRR.

    First, it is difficult to link changes in task difficulty to arousal level without demonstrating that the subjects did not change their strategy between easy and difficult task conditions by, for example, looking directly at the more difficult targets instead of maintaining central fixation as the task required. Without this control, the changes reported in TRRs could be attributed to changes in eye movements and the concomitant changes in the the visual field, especially given measurements were made in visual cortex.

    We have analyzed observers’ eye movements to rule out this possibility. Please see the new section of the results: “An eye-movement-evoked artifact could not account for task-related fMRI responses."

    In the same vein, a more detailed or explicit differentiation between the stimulus or attention-evoked hemodynamic response and the TRR is necessary to help the reader evaluate the TRR without simultaneous eye tracking to remove trials where the visual field may have changed.

    Please see the new section of the supplement titled “Stimulus-evoked activity during the localizer” and the new supplementary figures S4 and S5.

    Given arousal is a loosely defined cognitive phenomenon, physiologic arousal markers (ex. pupil, heart rate, respiration) are commonly used to track changes in arousal level, as is the case in this work. The evidence in this work that arousal level changed between task conditions (ex. difficult and easy trials) requires a more detailed analysis to control for the large number of variables and determine the effects that survive. While an accompanying data set showed changes in pupil diameter in a manner consistent with arousal changes during the task, this data was recorded in a separate experiment. This does provide a source of eye movement data for potential control analysis.

    We have eye data from the pupillometry recordings outside the scanner. We have reanalyzed these data, comparing the variance in eye position between conditions. We did not find any statistically significant differences in this measure of fixation stability between conditions.

    If the widespread BOLD responses were luminance-evoked, i.e., caused by eye movements changing the pattern of retinal stimulation, we would have expected to observe the strongest responses in V1 at eccentricities close to the representation of the screen edge. To test this hypothesis, we analyzed the BOLD signal amplitude across the representation of the visual field, but did not observe larger response amplitudes in the cortical locations corresponding to the edge of the screen.

    Lastly, the authors speculate about the origin of the TRR by comparing its magnitude and modulation in different task conditions across different levels of the visual cortical hierarchy (V1 vs. V2 vs V3). A direct statistical comparison of these effects would be necessary to convincingly demonstrate differences in the TRR across visual regions.

    We have performed this comparison and reported the results.

    Reviewer #2:

    The discussion suggests that this relates to a LC-NE arousal process. The connection is suggested by the data, but further work would be needed to cement this idea. The data are interesting, and a good window into further understanding of this effect.

    In the discussion line 330, they suggest that the TRR should be separately modeled and removed from fMRI data in preprocessing. While the authors have convinced this reader that the TRR is likely related to arousal, it is far from clear that this means that this effect should be removed from fMRI data in preprocessing. Many arousal effects exist naturally in fMRI data, and in brain activity in general. Many arousal effects are observable in spiking and LFPs. Since no spiking or LFPs were measured here, we don't know whether this signal is or is not related to spiking or LFPs (though some data from monkeys suggests a similar signal is hemodynamic only, it would take more convincing that the current TRRs arise from the same process as the previously reported primate literature).

    Agreed. We do not suggest that the TRR should be removed from fMRI data in all or even most scenarios. It can be removed if one has a specific reason to do so, e.g. separating a mixture of stimulus-evoked and task-related responses. We have revised our statement on this to clarify.

    Reviewer #3:

    However, a weakness of the paper is that the authors do not pursue the computational/functional significance, nor the biological drivers, of TRRs. For instance, linking TRRs to an explicit model of decision-making (beyond showing they covary with RTs and lapses), or further discussing their potential link to widespread arousal and movement variables in rodent calcium imaging and ephys data, would strongly increase the interest from those beyond the visual fMRI community.

    We have now discussed how our findings relate to a relevant computational model that links arousal and decision-making. And we have discussed how merging linear modeling of the TRR with computational models of decision-making may be a fruitful future direction for research. We have also discussed the possible links to widespread arousal-related cortical responses observed in rodent calcium imaging and ephys data.

  3. eLife

    Evaluation Summary:

    This work will be of general interest to those using hemodynamic imaging, such as fMRI, to study the brain. A hemodynamic signature that is modulated by arousal level changes on a trial-to-trial basis, such as those evoked by a difficult task, would both provide insight into arousal influences on cortical activity and characterize a prominent signal in hemodynamic data that is rarely considered. Overall, the data and analyses provide support for this idea, but would benefit from additional analysis, controls, and a better framework for integrating this work with the existing literature and mechanistic understanding of arousal, neural activity, and behavior.

    (This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #2 agreed to share their name with the authors.)

  4. eLife

    Reviewer #1 (Public Review):

    Burlingham and colleagues investigated how a task-related hemodynamic response in visual cortex, measured with fMRI-BOLD, varied with task difficulty and behavioral performance in humans during a visual orientation discrimination task. In prior work from this group (Roth et al. 2020, PLOS) and others (Cardoso et al. 2012, Nat. Neuro; Cardoso et al. 2019, PLOS), this "task-related response" (TRR) was shown to be distinct from a stimulus or attention-evoked response, modulated by reward size, and linked to physiologic arousal markers. Unique from prior work that centered on trial-averaged responses, a major aim of this work was to use a general linear mixed model (GLMM) to estimate TRRs at the single trial level. The authors showed TRRs were higher in amplitude and more precise in timing for more difficult trials and trials where the subject made an error or responded faster, situations consistent with elevated arousal. They propose arousal as an underlying driver for TRRs based on similar trends in physiologic metrics, such as pupil, respiration and heart rate. Additionally, they found evidence that TRRs, and the effects of task difficulty, were spatially diffuse (also present in V2 and V3), but weaker in magnitude higher in the visual hierarchy.

    Strengths:

    The task-related response (TRR) is a poorly understood hemodynamic signature that has been reported in the literature. Given its magnitude is similar to that of the stimulus-evoked hemodynamic response, understanding and accounting for TRRs will be critically useful for the field. This work takes a step forward in characterizing TRRs on a trial-by-trial basis with their use of a GLMM.

    Specifically, TRRs were modeled as the sum of the hemodynamic responses to trial onset, time on task, and button press. This model allowed Burlingham et al. to develop hypotheses to explain complex features of their results. For example, the effect of reaction time on the trial TRR would not have been easily captured by one variable, but rather the summed influence of the time the subject spent on the trial and the time the subject took to respond. Their GLMM allowed them to simulate fMRI activity in response to each input independently to show that the combined inputs best explained the experimental results. This modeling approach was a key strength of this work and future work on TRRs will greatly benefit from utilizing a similar approach. Even studies that do not aim to focus on studying the TRR can make use of this strategy to remove or account for this hemodynamic signature in the data.

    Additionally, this work evaluated changes in the TRR within the framework of arousal level changes related to behavioral performance and task difficulty. Identifying mechanisms by which arousal influences brain activity is of significant interest to the field of cognitive neuroscience. The task paradigm in this work evoked changes in arousal level by manipulating task difficulty and then assessed these changes using behavioral and physiologic markers of arousal. Their results provide additional support to existing literature that TRRs could be a cortical arousal signature.

    Weaknesses:

    While the implications are compelling, a few other controls and analyses would better establish the link between arousal level and the TRR.

    First, it is difficult to link changes in task difficulty to arousal level without demonstrating that the subjects did not change their strategy between easy and difficult task conditions by, for example, looking directly at the more difficult targets instead of maintaining central fixation as the task required. Without this control, the changes reported in TRRs could be attributed to changes in eye movements and the concomitant changes in the the visual field, especially given measurements were made in visual cortex. In the same vein, a more detailed or explicit differentiation between the stimulus or attention-evoked hemodynamic response and the TRR is necessary to help the reader evaluate the TRR without simultaneous eye tracking to remove trials where the visual field may have changed.

    Given arousal is a loosely defined cognitive phenomenon, physiologic arousal markers (ex. pupil, heart rate, respiration) are commonly used to track changes in arousal level, as is the case in this work. The evidence in this work that arousal level changed between task conditions (ex. difficult and easy trials) requires a more detailed analysis to control for the large number of variables and determine the effects that survive. While an accompanying data set showed changes in pupil diameter in a manner consistent with arousal changes during the task, this data was recorded in a separate experiment. This does provide a source of eye movement data for potential control analysis.

    Lastly, the authors speculate about the origin of the TRR by comparing its magnitude and modulation in different task conditions across different levels of the visual cortical hierarchy (V1 vs. V2 vs V3). A direct statistical comparison of these effects would be necessary to convincingly demonstrate differences in the TRR across visual regions.

  5. eLife

    Reviewer #2 (Public Review):

    The main results for this paper come from an fMRI study in 9 participants. Stimuli were Gabor patches, presented in the right lower visual field. The participants' response identifying the orientation of the Gabor was identified as correct or incorrect with a tone at the time of a button press. Stimuli were presented with a 15 s inter-stimulus interval to allow for examination of the hemodynamic response shape after each stimulus.

    "TRRs" were defined as responses time locked to these stimuli but in the ipsilateral visual field.

    The authors find that TRR magnitude (defined both based on Fourier amplitude and by mean % change) is strong in V1 (and may be localized to the superior bank). These responses are modulated by correctness (Figure 2B, effect for Easy trial only), and by reaction time (both easy and hard correct and incorrect trials) (Figure 2D). Interestingly, the effect of reaction time was opposite in the Hard vs Easy trials. The authors used a GLMM model to show that task onset, button press and time on task all contribute to the signal.

    These TRRs were biggest in V1, and smaller in V2 and V3.

    This scaling with reaction time fit a model incorporating trial onset, button press, as well as time on task (reaction time).
    The reviewers also did a similar experiment in 5 participants examining physiological responses outside the scanner, with a 3.5 s inter-trial interval. Accuracy and difficulty also had similar effects on these metrics. This suggests a hypothesis that the TRR effect may be driven by something similar to the physiological responses.

    Taken together these results suggest that there is trial-driven hemodynamic signal in V1 that covaries with behavior in a way that is consistent with arousal. The discussion suggests that this relates to a LC-NE arousal process. The connection is suggested by the data, but further work would be needed to cement this idea. The data are interesting, and a good window into further understanding of this effect.

    In the discussion line 330, they suggest that the TRR should be separately modeled and removed from fMRI data in preprocessing. While the authors have convinced this reader that the TRR is likely related to arousal, it is far from clear that this means that this effect should be removed from fMRI data in preprocessing. Many arousal effects exist naturally in fMRI data, and in brain activity in general. Many arousal effects are observable in spiking and LFPs. Since no spiking or LFPs were measured here, we don't know whether this signal is or is not related to spiking or LFPs (though some data from monkeys suggests a similar signal is hemodynamic only, it would take more convincing that the current TRRs arise from the same process as the previously reported primate literature).

  6. eLife

    Reviewer #3 (Public Review):

    Burlingham et al. extend their previous work on fMRI signals ('task-related responses' or TRRs) that are measured in visual cortex in response to task events, but which are spatially more widespread - in this case, to ipsilateral V1, V2 and V4, which receive input from the visual hemifield where no stimulus is presented. They show that these TRRs covary with a number of task and behavioral factors on a trial-by-trial basis. Similar patterns are observed for psychophysiological measures: pupil dilation, heart rate and breathing.

    This paper characterizes an interesting signal that is especially important to take into account when studying visual fMRI responses. The modeling and quantification is rigorous, and the link to central arousal systems is very promising. However, a weakness of the paper is that the authors do not pursue the computational/functional significance, nor the biological drivers, of TRRs. For instance, linking TRRs to an explicit model of decision-making (beyond showing they covary with RTs and lapses), or further discussing their potential link to widespread arousal and movement variables in rodent calcium imaging and ephys data, would strongly increase the interest from those beyond the visual fMRI community.

  7. Version published to 10.1101/2021.07.14.452398v1 on bioRxiv