Nigrostriatal Dopamine Signals Sequence-Specific Action-Outcome Prediction Errors
Curation statements for this article:-
Curated by eLife
Evaluation Summary:
This manuscript is of interest to neuroscientists conducting reward learning and basal ganglia research. It combines optogenetics and dopamine recordings to demonstrate that dopamine release in the dorsal striatum is smaller following self-stimulation than unpredicted stimulation of dopamine neurons. These results build on similar findings recently shown for the ventral striatum. Further development of the underlying mechanism or the behavioral significance would broaden the scope of the paper.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 agreed to share their name with the authors.)
This article has been Reviewed by the following groups
Discuss this preprint
Start a discussion What are Sciety discussions?Listed in
- Evaluated articles (eLife)
Abstract
Dopamine has been suggested to encode cue-reward prediction errors during Pavlovian conditioning. While this theory has been widely applied to reinforcement learning concerning instrumental actions, whether dopamine represents action-outcome prediction errors and how it controls sequential behavior remain largely unknown. Here, by training mice to perform optogenetic intracranial self-stimulation, we examined how self-initiated goal-directed behavior influences nigrostriatal dopamine transmission during single as well as sequential instrumental actions. We found that dopamine release evoked by direct optogenetic stimulation was dramatically reduced when delivered as the consequence of the animal’s own action, relative to non-contingent passive stimulation. This action-induced dopamine suppression was specific to the reinforced action, temporally restricted to counteract the expected outcome, and exhibited sequence-selectivity consistent with hierarchical control of sequential behavior. Together these findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations, with fundamental implications for reinforcement learning and instrumental behavior in health and disease.
Article activity feed
-
Evaluation Summary:
This manuscript is of interest to neuroscientists conducting reward learning and basal ganglia research. It combines optogenetics and dopamine recordings to demonstrate that dopamine release in the dorsal striatum is smaller following self-stimulation than unpredicted stimulation of dopamine neurons. These results build on similar findings recently shown for the ventral striatum. Further development of the underlying mechanism or the behavioral significance would broaden the scope of the paper.
(This preprint has been reviewed by eLife. We include the public reviews from the reviewers here; the authors also receive private feedback with suggested changes to the manuscript. Reviewer #1 agreed to share their name with the authors.)
-
Reviewer #1 (Public Review):
Hollon et al. characterized dopamine release in mice performing optogenetic intracranial self-stimulation (opto-ICSS). In this paradigm, a lever press resulted in an optogenetic stimulation of dopamine neurons in the substantia nigra pars compacta (SNc). The authors monitored dopamine concentrations in the dorsomedial striatum using fast-scan cyclic voltammetry (FSCV). The authors show that dopamine release evoked by opto-ICSS is reduced when stimulation comes as a result of the animal's action, relative to non-contingent stimulation. This reduction is sensitive to the action sequence preceding the rewarded action, arguing for a role of dopamine prediction errors in the hierarchical control of behavior. The authors conclude that "these findings demonstrate that nigrostriatal dopamine signals …
Reviewer #1 (Public Review):
Hollon et al. characterized dopamine release in mice performing optogenetic intracranial self-stimulation (opto-ICSS). In this paradigm, a lever press resulted in an optogenetic stimulation of dopamine neurons in the substantia nigra pars compacta (SNc). The authors monitored dopamine concentrations in the dorsomedial striatum using fast-scan cyclic voltammetry (FSCV). The authors show that dopamine release evoked by opto-ICSS is reduced when stimulation comes as a result of the animal's action, relative to non-contingent stimulation. This reduction is sensitive to the action sequence preceding the rewarded action, arguing for a role of dopamine prediction errors in the hierarchical control of behavior. The authors conclude that "these findings demonstrate that nigrostriatal dopamine signals sequence-specific prediction errors in action-outcome associations".
Overall, the experiments (e.g. yoked stimulation control) are generally well-designed and the results are very interesting, if not completely novel. The authors cite two old papers (Garris et al., 1999; Kilpatrick et al., 2000) that used electrical ICSS combined with FSCV to report the same main finding as this paper: that evoked dopamine is dramatically reduced when it comes as the (predictable) result of an action. More recent studies have used optogenetic stimulation in combination with FSCV (Owesson-White et al., 2016; Rodeberg et al., 2016) or dopamine sensor (Covey and Cheer, 2019) in the nucleus accumbens (NAc). Another important study, which is not cited in the manuscript, is Kremer et al. (2020) which recorded from opto-tagged dopamine neurons in the ventral tegmental area (VTA) during a self-paced, operant task (although a cue was used in the middle of a trial) and demonstrated aspects of reward prediction error coding. Although there are some technical differences (e.g. the use of yoked stimulation delivery schedule), these previous studies overlap with some of the experiments and conclusions of the present study.
Considering these previous studies, the novelty of the present study lies in two points. First, dopamine concentrations were monitored in the dorsomedial striatum, where dopamine signals were less characterized than in the well-studied NAc. Second, more importantly, the authors demonstrate that the expectation-dependent suppression is sensitive to the relative position within a sequence (Fig. 4I-P). This finding, along with the lack of suppression after the first lever press (Fig. 4A-H), suggests "sequence-level hierarchical control over instrumental behavior." The manuscript is generally well-written, and the data are presented clearly. Although the manuscript contains various interesting observations, there are many claims that are not directly supported by the presented data. These concerns need to be addressed before publication.
1. The title states that "Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors". The authors make similar conclusions multiple times in the manuscript. However, the current experiment involves artificial stimulation. This is particularly problematic as at least some previous studies have indicated that not all dopamine neurons are activated by reward (e.g. Howe and Dombeck, 2016; da Silva et al., 2018). Although optogenetic stimulation provides some technical advantages, it is unclear whether the authors' findings apply to more natural conditions. In order to show that nigrostriatal dopamine signals sequence-specific action-outcome prediction errors, it is essential to examine dopamine signals using a natural reward. Without such a demonstration, the conclusion should be revised to reflect what are directly demonstrated by the presented data.
2. Another related issue is the artificial nature of this stimulation. For instance, Coddington and Dudman (2018) showed that reward-matched stimulation (in their setup, 1 mW, 150 ms duration, 10 ms pulses at 30 Hz) can have very different effects than artificially large stimulation. The stimulation parameters used here (5 mW, 1000 ms duration, 10 ms pulses at 50 Hz) are expected to drive dopamine neurons outside the physiological range. To address this issue, the authors should carefully discuss the magnitude and the time course of dopamine response evoked by the stimulation based on the data. Ideally, the dopamine dynamics should be compared directly between those evoked optogenetically and those evoked by a natural reward. If such a data does not exist, the authors could use carefully-calibrated estimates of dopamine concentration between these conditions collected in separate experiments. In such a case, how dopamine concentrations are calibrated needs to be reported.
3. Line 103: "Thus, in this entirely within-subject design, we recorded at the same striatal location with the same temporal sequence of stimulations across both phases of the session, delivered to the same site within the SNc using identical optogenetic stimulation parameters to directly depolarize these nigrostriatal dopamine neurons." Although this statement is true, it is not guaranteed that the animal's location or general motivational states are controlled across the control and experimental conditions. Accordingly, or in addition, sensory experiences (inputs) can be very different.
4. One of the novel aspects of this study is monitoring dopamine concentrations in the dorsomedial striatum. However, interpretation of FSCV signals might not be very straightforward if the region contains noradrenergic or serotonergic inputs. How do the authors control for that? Although the use of optogenetic stimulation would help address this issue, it is not entirely obvious whether the observed signals contains only dopamine signals. More justifications of signal specificity would be very helpful, especially for those who are not familiar with the technique (FSCV).
5. In Figure 1F, cyclic voltammograms show very different patterns between the self-stimulation and passive playback conditions. Can the authors be sure that the signals represent dopamine?
6. Line 181. "Furthermore, because the sensory feedback from pressing the Active and Inactive levers is rather similar to the animals, these results confirmed that the dopamine inhibition results from the expectations associated with specific self-initiated, goal-directed action but not simply a conditioned sensory cue". This statement appears to be a little overstatement. First, the visual inputs at Active and Inactive levers are not identical because of the different geometric locations in the operant chamber. This experiment alone does not allow the authors to make the above claim (the later experiment testing the effect of action sequence directly addresses this issue). Furthermore, this experiment does not demonstrate the necessity of "self-initiated" nor "goal-directed". The above statement gives the impression that these two factors are important. These discussions should be made more carefully throughout the manuscript.
7. Related to the above issue, a movement or sequence of movements would inevitably induce proprioceptive and mechanical sensations, and the sequence thereof. Furthermore, pressing the left and right lever would be accompanied by different sensory experiences (the visual scenes are different, and perhaps the animal's posture, and the resulting proprioceptive signals, may be different too). Therefore, strictly speaking, whether the suppression of dopamine response was induced by "action" or by sensory inputs cannot be separated by the presented data. It appears that the data strongly support the importance of sequence (or history) but does not distinguish whether it is the sequence of actions or sensory experiences, or a combination of them. Considering these issues, the distinction between "cue-reward prediction errors" and "action-outcome prediction errors" appears to be difficult, at least from the presented data. It would be important to discuss this point more carefully.
-
Reviewer #2 (Public Review):
The study examines dopamine transients when a mouse is approaching a lever to optogentically self-stimulate SNc neurons and how sequences influence the nigrostriatal dopamine response to the consequence of these actions.
The claim is that a goal directed action partially reduces the optogenetically evoked DA transients.
The study then characterises the temporal dynamics, modulation by reward omission and during action sequences, but fall short of elucidating the underlying mechanism (a postsynaptic effect on stimulation efficacy is hinted) or establishing causalities with elements of the behavior.
-
Reviewer #3 (Public Review):
Hollon and colleagues record dopamine signaling in the dorsal striatum using fast scan voltammetry while mice engage in optogenetic self stimulation of nigrostriatal dopamine neurons. They find that optically evoked dopamine release following goal directed self-stimulation is smaller, relative to unpredicted optical stimulation, indicating that goal directed actions can contain reward prediction errors to track expected nigrostriatal dopamine signaling, independent of additional sensory processing typical of natural reward learning. They also show that this expected-mediated inhibition occurs when mice engage in a two-step sequence of actions to obtain optogenetic stimulation. I have a few comments about these results in the context of other recently published data, and some questions about the analysis.
The …
Reviewer #3 (Public Review):
Hollon and colleagues record dopamine signaling in the dorsal striatum using fast scan voltammetry while mice engage in optogenetic self stimulation of nigrostriatal dopamine neurons. They find that optically evoked dopamine release following goal directed self-stimulation is smaller, relative to unpredicted optical stimulation, indicating that goal directed actions can contain reward prediction errors to track expected nigrostriatal dopamine signaling, independent of additional sensory processing typical of natural reward learning. They also show that this expected-mediated inhibition occurs when mice engage in a two-step sequence of actions to obtain optogenetic stimulation. I have a few comments about these results in the context of other recently published data, and some questions about the analysis.
The major take home from this paper is that optically evoked dopamine signals in the dorsomedial striatum diminish when they are under an animal's control. This expectation-based inhibition of dopamine is a signature of natural reward learning and it's really interesting that the same phenomenon occurs when dopamine neurons are artificially activated with optogenetics. My primary thought is that it is not clear what new information these studies demonstrate beyond some recently published findings. Covey and Cheer 2019 demonstrate a similar effect in the nucleus accumbens - optogenetically evoked dopamine diminishes as it become expected during ICSS. Older FSCV papers also demonstrated this goal-directed inhibition effect for electrical self stimulation - which, while different from opto ICSS, I think in this case it's effectively the same.
Here we see that also applies to the dorsomedial striatum, which is an important thing to demonstrate, especially in light of lots of recent data showing heterogeneous dopamine circuit encoding and function across striatal sub region. But it's unclear from the paper if there was an expectation that this process would be different in dorsal striatum? At the level of the cell body, at least, reward prediction error signatures are similar across dopamine neurons in the VTA and SNC. However, given that nigrostriatal and mesostriatal dopamine terminals have different characteristics (for example, dorsal striatum dopamine terminals have higher levels of DAT), different input connectivity, etc, you might expect heterogeneity. In light of that recent literature it is perhaps surprising therefore that a similar expected-mediated effect on optically-evoked dopamine holds in the dorsal striatum, but the current manuscript does not give insight into why that is.
-
