Abstract
Current models of the basal ganglia and dopamine neurons emphasize their role in reinforcement learning. However, the role of dopamine neurons in decision making is still unclear. We recorded from dopamine neurons in monkeys engaged in two types of trial: reference trials in an instructed-choice task and decision trials in a two-armed bandit decision task. We show that the activity of dopamine neurons in the decision setting is modulated according to the value of the upcoming action. Moreover, analysis of the probability matching strategy in the decision trials revealed that the dopamine population activity and not the reward during reference trials determines choice behavior. Because dopamine neurons do not have spatial or motor properties, we conclude that immediate decisions are likely to be generated elsewhere and conveyed to the dopamine neurons, which play a role in shaping long-term decision policy through dynamic modulation of the efficacy of basal ganglia synapses.
This is a preview of subscription content, access via your institution
Access options
Subscribe to this journal
Receive 12 print issues and online access
$209.00 per year
only $17.42 per issue
Buy this article
- Purchase on Springer Link
- Instant access to full article PDF
Prices may be subject to local taxes which are calculated during checkout
Similar content being viewed by others
References
Herrnstein, R.J. On the law of effect. J. Exp. Anal. Behav. 13, 243–266 (1970).
Tversky, A. & Kahneman, D. The framing of decisions and the psychology of choice. Science 211, 453–458 (1981).
Bayer, H.M. & Glimcher, P.W. Midbrain dopamine neurons encode a quantitative reward prediction error signal. Neuron 47, 129–141 (2005).
Sugrue, L.P., Corrado, G.S. & Newsome, W.T. Matching behavior and the representation of value in the parietal cortex. Science 304, 1782–1787 (2004).
Romo, R., Hernandez, A. & Zainos, A. Neuronal correlates of a perceptual decision in ventral premotor cortex. Neuron 41, 165–173 (2004).
Barraclough, D.J., Conroy, M.L. & Lee, D. Prefrontal cortex and decision making in a mixed-strategy game. Nat. Neurosci. 7, 404–410 (2004).
Samejima, K., Ueda, Y., Doya, K. & Kimura, M. Representation of action-specific reward values in the striatum. Science 310, 1337–1340 (2005).
Pagnoni, G., Zink, C.F., Montague, P.R. & Berns, G.S. Activity in human ventral striatum locked to errors of reward prediction. Nat. Neurosci. 5, 97–98 (2002).
Sutton, R.S. & Barto, A.G. Reinforcement Learning: an Introduction (MIT Press, Cambridge, Massachusetts, 1998).
Rummery, G.A. & Niranjan, M. On-line Q-learning using connectionist systems. Technical Report CUED/F-INFENG/TR 166 (Engineering Department, Cambridge University, Cambridge, UK, 1994).
Watkins, C.J.C.H. & Dayan, P. Q learning. Mach. Learn. 8, 279–292 (1992).
Dayan, P. & Balleine, B.W. Reward, motivation, and reinforcement learning. Neuron 36, 285–298 (2002).
Schultz, W., Dayan, P. & Montague, P.R. A neural substrate of prediction and reward. Science 275, 1593–1599 (1997).
Fiorillo, C.D., Tobler, P.N. & Schultz, W. Discrete coding of reward probability and uncertainty by dopamine neurons. Science 299, 1898–1902 (2003).
Satoh, T., Nakai, S., Sato, T. & Kimura, M. Correlated coding of motivation and outcome of decision by dopamine neurons. J. Neurosci. 23, 9913–9923 (2003).
Morris, G., Arkadir, D., Nevet, A., Vaadia, E. & Bergman, H. Coincident but distinct messages of midbrain dopamine and striatal tonically active neurons. Neuron 43, 133–143 (2004).
Nakahara, H., Itoh, H., Kawagoe, R., Takikawa, Y. & Hikosaka, O. Dopamine neurons can represent context-dependent prediction error. Neuron 41, 269–280 (2004).
Pan, W.X., Schmidt, R., Wickens, J.R. & Hyland, B.I. Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network. J. Neurosci. 25, 6235–6242 (2005).
Gurney, K., Prescott, T.J., Wickens, J.R. & Redgrave, P. Computational models of the basal ganglia: from robots to membranes. Trends Neurosci. 27, 453–459 (2004).
Mink, J.W. The basal ganglia: focused selection and inhibition of competing motor programs. Prog. Neurobiol. 50, 381–425 (1996).
Bar-Gad, I., Morris, G. & Bergman, H. Information processing, dimensionality reduction and reinforcement learning in the basal ganglia. Prog. Neurobiol. 71, 439–473 (2003).
Barto, A.G. Adaptive critics and the basal ganglia. in Models of Information Processing in the Basal Ganglia (eds. Houk, J.C., Davis, J.L. & Beiser, D.G.) 215–232 (MIT Press, Cambridge, Massachusetts, 1995).
Reynolds, J.N., Hyland, B.I. & Wickens, J.R. A cellular mechanism of reward-related learning. Nature 413, 67–70 (2001).
McClure, S.M., Daw, N.D. & Montague, P.R. A computational substrate for incentive salience. Trends Neurosci. 26, 423–428 (2003).
Tobler, P.N., Fiorillo, C.D. & Schultz, W. Adaptive coding of reward value by dopamine neurons. Science 307, 1642–1645 (2005).
Wolford, G., Miller, M.B. & Gazzaniga, M. The left hemisphere's role in hypothesis formation. J. Neurosci. 20, RC64 (2000).
Vulkan, N. An economist's perspective on probability matching. J. Econ. Surv. 14, 101–118 (2000).
Lau, B. & Glimcher, P.W. Dynamic response-by-response models of matching behavior in rhesus monkeys. J. Exp. Anal. Behav. 84, 555–579 (2005).
Dommett, E. et al. How visual stimuli activate dopaminergic neurons at short latency. Science 307, 1476–1479 (2005).
Wise, R.A. Dopamine, learning and motivation. Nat. Rev. Neurosci. 5, 483–494 (2004).
Cragg, S.J. & Rice, M.E. DAncing past the DAT at a DA synapse. Trends Neurosci. 27, 270–277 (2004).
Wickens, J.R. & Arbuthnot, G.W. Structural and functional interactions in the striatum at the receptor level. in Dopamine (eds. Dunnett, S.B., Bentivoglio, M., Bjorklund, A. & Hokfelt, T.) 199–236 (Elsevier, Amsterdam, 2005).
Waelti, P., Dickinson, A. & Schultz, W. Dopamine responses comply with basic assumptions of formal learning theory. Nature 412, 43–48 (2001).
Szabo, J. & Cowan, W.M. A stereotaxic atlas of the brain of the cynomolgus monkey (Macaca fascicularis). J. Comp. Neurol. 222, 265–300 (1984).
Martin, R.F. & Bowden, D.M. Primate Brain Maps: Structure of the Macaque Brain (Elsevier Science, Amsterdam, 2000).
Grace, A.A. & Bunney, B.S. Intracellular and extracellular electrophysiology of nigral dopaminergic neurons–1. Identification and characterization. Neuroscience 10, 301–315 (1983).
Ungless, M.A., Magill, P.J. & Bolam, J.P. Uniform inhibition of dopamine neurons in the ventral tegmental area by aversive stimuli. Science 303, 2040–2042 (2004).
Hollerman, J.R. & Schultz, W. Dopamine neurons report an error in the temporal prediction of reward during learning. Nat. Neurosci. 1, 304–309 (1998).
Nevet, A., Morris, G., Saban, G., Fainstein, N. & Bergman, H. Discharge rate of substantia nigra pars reticulata neurons is reduced in non-parkinsonian monkeys with apomorphine-induced orofacial dyskinesia. J. Neurophysiol. 92, 1973–1981 (2004).
Sokal, R.R. & Rohlf, F.J. Biometry (W.H. Freeman & Co., New York, 1981).
Acknowledgements
We thank Y. Prut, Y. Engel, Y. Ritov, N. Daw, Y. Niv and R. Paz for fruitful discussions and comments on earlier versions of this manuscript; and G. Goelman and V. Sharkansky for technical assistance. This study was partly supported by a Center of Excellence grant administered by the Israel Science Foundation (ISF) and by a 'Fighting against Parkinson' grant administered by the Netherlands Friends of the Hebrew University (HUNA). G.M. was supported by a Horowitz fellowship.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing financial interests.
Supplementary information
Supplementary Fig. 1
Choice triggered averaging (CTA) functions in decision trials. (PDF 321 kb)
Supplementary Fig. 2
Conditioning stimulus responses according to choice value. (PDF 168 kb)
Rights and permissions
About this article
Cite this article
Morris, G., Nevet, A., Arkadir, D. et al. Midbrain dopamine neurons encode decisions for future action. Nat Neurosci 9, 1057–1063 (2006). https://doi.org/10.1038/nn1743
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1038/nn1743
This article is cited by
-
Striatal dopamine signals reflect perceived cue–action–outcome associations in mice
Nature Neuroscience (2024)
-
Navigating for reward
Nature Reviews Neuroscience (2021)
-
Beyond dichotomies in reinforcement learning
Nature Reviews Neuroscience (2020)
-
Dissociable dopamine dynamics for learning and motivation
Nature (2019)