Review
Feature Review
The neural mechanisms of inter-temporal decision-making: understanding variability

https://doi.org/10.1016/j.tics.2011.03.002Get rights and content

Humans and animals prefer immediate over delayed rewards (delay discounting). This preference for smaller-but-sooner over larger-but-later rewards shows substantial interindividual variability in healthy subjects. Moreover, a strong bias towards immediate reinforcement characterizes many psychiatric conditions such as addiction and attention-deficit hyperactivity disorder. We discuss the neural mechanisms underlying delay discounting and describe how interindividual variability (trait effects) in the neural instantiation of subprocesses of delay discounting (such as reward valuation, cognitive control and prospection) contributes to differences in behaviour. We next discuss different interventions that can partially remedy impulsive decision-making (state effects). Although the precise neural mechanisms underlying many of these modulating influences are only beginning to be unravelled, they point towards novel treatment approaches for disorders of impulse control.

Section snippets

The subjective nature of preferences

Experience tells us that preferences are subjective. Some people are patient, others impatient, some take risks and others tend to avoid them. In recent years, many areas of decision science, including psychology, behavioural economics, psychiatry and cognitive and systems neuroscience, have adopted approaches that focus on subjective aspects of choice and valuation. One area of research that has been extremely fruitful in the study of subjective choice and valuation is intertemporal decision

Standard models

Computational models of DD aim to develop functions that capture the relationship between temporal proximity and subjective value (Figure 1a). Two models have dominated the field of temporal discounting for the last decades and both include a single free parameter, the discount rate k. The decay of subjective value (SV) over time is modelled as an exponential (i.e. SV=AekD [2]) or hyperbolic (i.e. SV=A/(1+kD) [3]) function of delay, where A is the objective amount of the reward, D is the delay

The trait–state distinction

An important question that arises in decision research is whether certain differences observed in choice and valuation (e.g. between different individuals or experimental conditions) are caused by state or trait differences. In particular with respect to DD, it has recently been suggested that both factors might affect decision-making [21]. In this view, preferences are flexible and dependent on the decision context or current requirements of the organism. These factors might thus induce a

The cognitive neuroscience of DD: trait effects

Recent theoretical accounts distinguish between at least two general processing stages in decision-making [49]: valuation, which is the neural computation and representation of the subjective values of available decision options, and choice, which comprises processes leading to and supporting action selection. Following this taxonomy, we first focus on the neural mechanisms underlying reward valuation processes in DD. We then address the role of processes related to conflict monitoring and

Domain-general and domain-specific valuation

Overwhelming evidence implicates the ventral striatum (VS) and orbitofrontal cortex (OFC), in particular its ventromedial part (often referred to as ventromedial prefrontal cortex, vmPFC, or medial OFC) in the representation of the incentive value of a broad range of different classes of rewards 49, 50, 51, 52. The VS is a projection region of dopaminergic neurons in the substantia nigra (SN) and the ventral tegmental area (VTA), which have been implicated in reinforcement learning and reward

Conflict monitoring, strategy adaptation and the anterior cingulate cortex

In addition to effects of subjective reward value, decision-making is affected by another important variable, decision conflict. Decisions are difficult when options are of similar value, whereas decisions are easier when option values are clearly different. The degree of decision conflict in such situations is correlated with activity in prefrontal control regions, in particular the anterior cingulate cortex (ACC) [77]. In intertemporal choice, this effect is typically examined by comparing

MTL contributions to DD

Two regions in the MTL, the hippocampus and the amygdala, have repeatedly been implicated in DD, although their precise contributions are poorly understood. Damage to the hippocampus increases DD in rats 88, 89, 90, although it is unclear whether this constitutes a selective impairment in DD or generalizes to other forms of cost–benefit decision-making. Amygdala damage in rodents, conversely, is known to impair performance in a range of cost–benefit choice tasks, including probability

Interim conclusions

We have shown that at least three distinct networks contribute to intertemporal decision-making. A valuation network comprising vmPFC, mOFC, ventral striatum and PCC represents the subjective discounted value of future rewards. The lateral PFC and ACC are involved in DD, predominantly through their role in cognitive control, conflict monitoring and strategy adaptation. Finally, although still poorly understood, MTL regions including the hippocampus might contribute to DD through representing

Contextual modulations in DD: state effects

In light of the consistent association of DD with substance abuse and addiction, it is of high clinical relevance to identify mechanisms or interventions that reduce impulsive discounting. At the same time, a better understanding of modulating factors could enhance our understanding of the psychological (and neural) processes underlying DD. We address three basic types of state modulation that provide insight into the psychological and neural mechanisms of DD, as well as into the potential for

Interim conclusions

The findings summarized in the preceding sections show that DD is a considerably stable trait within individuals, but one that is also subject to modulation on various levels. Behavioural framing and context effects suggest an important role for mental representation of the decision problem: how subjects represent delay and/or outcome seems to be a crucial factor in the valuation of that outcome. The overall decision context might also activate learned behavioural patterns, which might play a

Concluding remarks

A basic model of the functional neuroanatomy of intertemporal choice is beginning to take shape. This model includes neural circuits that support different aspects of intertemporal decisions: vmPFC, VS and PCC are involved in the representation of subjective discounted value [49], and hyposensitivity of these regions might contribute to impulsive discounting in psychiatric conditions. PFC and ACC are part of a network that exerts cognitive control during decision-making, and in DD might bias

Acknowledgements

We thank Markus Staudinger and Sebastian Gluth for helpful comments on a previous version of this manuscript.

Glossary

Delay (or temporal) discounting
the phenomenon that agents typically devalue rewards as a function of the delay to their delivery.
Dynamic inconsistency
Consistent delay discounting entails that adding a common constant delay to the available options does not change which option is preferred by an individual. The fact that human preferences are typically not consistent in this fashion is referred to as dynamic inconsistency. For example, an individual may prefer 20€ in 1 week over 25€ in 2 weeks,

References (163)

  • T.A. Wills

    Novelty seeking, risk taking, and related constructs as predictors of adolescent substance use: an application of Cloninger's theory

    J. Subst. Abuse.

    (1994)
  • J.W. Kable et al.

    The neurobiology of decision: consensus and controversy

    Neuron

    (2009)
  • J. Peters et al.

    Neural representations of subjective reward value

    Behav. Brain Res.

    (2010)
  • M.L. Kringelbach et al.

    The functional neuroanatomy of the human orbitofrontal cortex: evidence from neuroimaging and neuropsychology

    Prog. Neurobiol.

    (2004)
  • G. Schoenbaum et al.

    Encoding predicted outcome and acquired value in orbitofrontal cortex during cue sampling depends upon input from basolateral amygdala

    Neuron

    (2003)
  • T. Kalenscher

    Single units in the pigeon brain integrate reward amount and time-to-reward in an impulsive choice task

    Curr. Biol.

    (2005)
  • B.J. Weber et al.

    The neural substrates of probabilistic and intertemporal decision making

    Brain Res.

    (2008)
  • E.K. Diekhof

    Functional neuroimaging of reward processing and decision-making: a review of aberrant motivational and affective processing in addiction and mood disorders

    Brain Res. Rev.

    (2008)
  • M. Buhler

    Nicotine dependence is characterized by disordered reward processing in a network driving motivation

    Biol. Psychiatry

    (2010)
  • J. Marco-Pallares

    Brain activations reflect individual discount rates in intertemporal choice

    Brain Res.

    (2010)
  • J.N. Rawlins

    The effects of delaying reward on choice preference in rats with hippocampal or selective septal lesions

    Behav. Brain Res.

    (1985)
  • M.A. van der Meer

    Triple dissociation of information processing in dorsal striatum, ventral striatum, and hippocampus on a learned spatial decision task

    Neuron

    (2010)
  • L. Green et al.

    A discounting framework for choice with delayed and probabilistic rewards

    Psychol. Bull.

    (2004)
  • P.A. Samuelson

    A note on measurement of utility

    Rev. Econ. Stud.

    (1937)
  • J.E. Mazur

    An adjusting procedure for studying delayed reinforcement

  • J. Myerson

    Area under the curve as a measure of discounting

    J. Exp. Anal. Behav.

    (2001)
  • S.M. McClure

    Separate neural systems value immediate and delayed monetary rewards

    Science

    (2004)
  • J. Myerson et al.

    Discounting of delayed rewards: models of individual choice

    J. Exp. Anal. Behav.

    (1995)
  • H. Rachlin

    Notes on discounting

    J. Exp. Anal. Behav.

    (2006)
  • G. Zauberman

    Discounting time and time discounting: subjective time perception and intertemporal preferences

    J. Marketing Res.

    (2009)
  • J.W. Kable et al.

    The neural correlates of subjective value during intertemporal choice

    Nat. Neurosci.

    (2007)
  • J. Peters et al.

    Overlapping and distinct neural systems code for subjective value during intertemporal and risky decision making

    J. Neurosci.

    (2009)
  • Y. Ohmura

    Three-month stability of delay and probability discounting measures

    Exp. Clin. Psychopharmacol.

    (2006)
  • A.P. Anokhin

    Heritability of delay discounting in adolescence: a longitudinal twin study

    Behav. Genet.

    (2011)
  • K.N. Kirby

    One-year temporal stability of delay-discount rates

    Psychon. Bull. Rev.

    (2009)
  • M.R. Dixon

    Contextual control of delay discounting by pathological gamblers

    J. Appl. Behav. Anal.

    (2006)
  • J.W. Kable et al.

    An “as soon as possible” effect in human intertemporal decision making: behavioral evidence and neural mechanisms

    J. Neurophysiol.

    (2010)
  • W.K. Bickel

    Behavioral and neuroeconomics of drug addiction: competing neural systems and temporal discounting processes

    Drug Alcohol Depend.

    (2007)
  • K.N. Kirby

    Heroin addicts have higher discount rates for delayed rewards than non-drug-using controls

    J. Exp. Psychol. Gen.

    (1999)
  • G.J. Madden

    Impulsive and self-control choices in opioid-dependent patients and non-drug-using control participants: drug and monetary rewards

    Exp. Clin. Psychopharmacol.

    (1997)
  • J.R. Monterosso

    Frontoparietal cortical activity of methamphetamine-dependent and comparison subjects performing a delay discounting task

    Hum. Brain Mapp.

    (2007)
  • W.K. Bickel

    Impulsivity and cigarette smoking: delay discounting in current, never, and ex-smokers

    Psychopharmacology (Berl.)

    (1999)
  • S.H. Mitchell

    Measures of impulsivity in cigarette smokers and non-smokers

    Psychopharmacology (Berl.)

    (1999)
  • J. Peters

    Lower ventral striatal activation during reward anticipation in adolescent smokers

    Am. J. Psychiatry

    (2011)
  • M.W. Johnson

    Moderate drug use and delay discounting: a comparison of heavy, light, and never smokers

    Exp. Clin. Psychopharmacol.

    (2007)
  • N.M. Petry

    Delay discounting of money and alcohol in actively using alcoholics, currently abstinent alcoholics, and controls

    Psychopharmacology (Berl.)

    (2001)
  • J.M. Mitchell

    Impulsive responding in alcoholics

    Alcohol Clin. Exp. Res.

    (2005)
  • N.M. Petry

    Pathological gamblers, with and without substance use disorders, discount delayed rewards at high rates

    J. Abnorm. Psychol.

    (2001)
  • M.R. Dixon

    Delay discounting by pathological gamblers

    J. Appl. Behav. Anal.

    (2003)
  • J. Reuter

    Pathological gambling is linked to reduced activation of the mesolimbic reward system

    Nat. Neurosci.

    (2005)
  • Cited by (463)

    View all citing articles on Scopus
    View full text