The neuropsychology of ventral prefrontal cortex: Decision-making and reversal learning

doi:10.1016/S0278-2626(03)00284-7

Brain and Cognition

Volume 55, Issue 1, June 2004, Pages 41-53

https://doi.org/10.1016/S0278-2626(03)00284-7 Get rights and content

Abstract

Converging evidence from human lesion, animal lesion, and human functional neuroimaging studies implicates overlapping neural circuitry in ventral prefrontal cortex in decision-making and reversal learning. The ascending 5-HT and dopamine neurotransmitter systems have a modulatory role in both processes. There is accumulating evidence that measures of decision-making and reversal learning may be useful as functional markers of ventral prefrontal cortex integrity in psychiatric and neurological disorders. Whilst existing measures of decision-making may have superior sensitivity, reversal learning may offer superior selectivity, particularly within prefrontal cortex. Effective decision-making on existing measures requires the ability to adapt behaviour on the basis of changes in emotional significance, and this may underlie the shared neural circuitry with reversal learning.

Introduction

Following brain injury to the orbitofrontal and ventromedial prefrontal cortex (PFC), human patients display gross alterations in social and emotional behaviour with largely preserved perception, language, memory, and even executive function (Bechara, Tranel, & Damasio, 2000; Damasio, 1994; Malloy, Bihrle, Duffy, & Cimino, 1993; Rolls, 1999). Characterisation of this profile using cognitive testing has been the target of considerable research, not least because the behaviour of patients with ventral prefrontal lesions resembles aspects of symptomatology seen in psychiatric conditions including psychopathy (Lapierre, Braun, & Hodgins, 1995) and substance abuse (Bechara & Damasio, 2002). Two cognitive domains have received particular attention in recent years: decision-making and reversal learning. The development of several measures of decision-making has stemmed largely from observations by Damasio, Bechara and colleagues, that patients with ventromedial prefrontal cortex damage are impaired in their ability to make successful everyday decisions regarding employment, relationships, and personal finances. Specifically, it has been proposed that these patients are unable to use past experiences to guide their ongoing decision-making (‘myopia for the future’) (Bechara et al., 2000; Damasio, 1994). Recent interest in reversal learning, in contrast, has developed from pre-clinical research over more than three decades demonstrating that rodents and non-human primates with lesions to the orbitofrontal cortex are unable to adapt their responding following changes in stimulus–reward contingencies (Butter, 1969; Jones & Mishkin, 1972).

The purpose of this article is to review converging evidence for the involvement of ventral prefrontal cortex in decision-making and reversal learning, from (1) human lesion studies, (2) animal lesion studies, and (3) human functional neuroimaging studies. Evidence for the contribution of the ascending 5-HT and dopamine neurotransmitter systems to these domains will also be described. Recent cognitive research in a number of clinical groups has begun to investigate the sensitivity and selectivity of decision-making and reversal learning deficits as indices of ventral prefrontal dysfunction.

Section snippets

The cognitive neuroscience of decision-making

Decision-making requires the evaluation of multiple response options, followed by the selection of the response considered optimal. Each response option may be characterised in terms of the reward and punishment outcomes with which it is associated. Response options may vary in terms of (1) the magnitude of reward and punishment, (2) the probability of receiving reward or punishment, and (3) the delay to reward or punishment. This framework provides scope for a range of decision-making

The cognitive neuroscience of reversal learning

Reversal learning involves the adaptation of behaviour according to changes in stimulus–reward contingencies, a capacity relevant to social and emotional behaviour (Rolls, 1999). It is exemplified by visual discrimination tasks where subjects must learn to respond according to the opposite, previously irrelevant, stimulus–reward pairing. Impaired reversal learning has been attributed to the loss of inhibitory control of affective responding (Dias, Robbins, & Roberts, 1996). As such, deficits in

The development of decision-making and reversal learning

The current interest in the neural mechanisms of decision-making and reversal learning has been based largely upon adult neuropsychological cases, and lesion studies in non-human primates, respectively. The development of these cognitive domains has only recently begun to be investigated. One issue (see also Bechara, this issue; Machado & Bachevalier, 2003) is whether the degree of cognitive dysfunction following developmental damage to orbitofrontal cortex is more, or less, severe than that

Synthesis

The ability to update and correct behaviour on the basis of changes in emotional significance is critical to performance on both reversal learning and decision-making tasks (see also Rolls, 1999). Reversal learning by definition involves a shift of responding from a stimulus that is no longer rewarded, to a previously unrewarded stimulus. In the context of the decision-making measures discussed above, the need for behavioural adaptation is more subtle and has previously been overlooked. Yet

Acknowledgments

This work was supported by a Wellcome Trust programme grant to T.W.R., B.J. Everitt, A.C. Roberts, & B.J. Sahakian, and completed within the Medical Research Council Centre for Behavioural and Clinical Neuroscience.

References (113)

L.E. Annett et al.
The effects of ibotenic acid lesions of the nucleus accumbens on spatial learning and extinction in the rat
Behavioral Brain Research
(1989)
A.F. Arnsten
Catecholamine modulation of prefrontal cortical cognitive function
Trends in Cognitive Sciences
(1998)
A.F. Arnsten et al.
The effects of 5-HT3 receptor antagonists on cognitive performance in aged monkeys
Neurobiology of Aging
(1997)
J.M. Barnes et al.
The effects of ondansetron, a 5-HT3 receptor antagonist, on cognition in rodents and primates
Pharmacology and Biochemistry of Behavior
(1990)
A. Bechara et al.
Decision-making and addiction (part I): Impaired activation of somatic states in substance dependent individuals when pondering decisions with negative future consequences
Neuropsychologia
(2002)
A. Bechara et al.
Insensitivity to future consequences following damage to human prefrontal cortex
Cognition
(1994)
C. Butter
Perseveration in extinction and in discrimination reversal tasks following selective frontal ablations in macaca mulatta
Physiology and Behavior
(1969)
L. Clark et al.
The contributions of lesion laterality and lesion volume to decision-making impairment following frontal lobe damage
Neuropsychologia
(2003)
L. Clark et al.
Decision-making deficits in drug addiction
Trends in Cognitive Sciences
(2002)
R. Cools et al.
l-Dopa medication remediates cognitive inflexibility, but increases impulsivity in patients with Parkinson’s disease
Neuropsychologia
(2003)

V. Czernecki et al.

Motivation, reward, and Parkinson’s disease: Influence of dopatherapy

Neuropsychologia

(2002)

A.R. Damasio et al.

Individuals with sociopathic behavior caused by frontal damage fail to respond autonomically to social stimuli

Behavioral Brain Research

(1990)

I. Daum et al.

T-maze discrimination and reversal learning after unilateral temporal or frontal lobe lesions in man

Cortex

(1991)

H. de Wit et al.

Acute administration of d-amphetamine decreases impulsivity in healthy volunteers

Neuropsychopharmacology

(2002)

A.M. Domeney et al.

The effect of ondansetron on cognitive performance in the marmoset

Pharmacology, Biochemistry and Behavior

(1991)

J.J. Downes et al.

Impaired extra-dimensional shift performance in medicated and unmedicated Parkinson’s disease: Evidence for a specific attentional dysfunction

Neuropsychologia

(1989)

M. Ernst et al.

Decision-making in a risk-taking task: A PET study

Neuropsychopharmacology

(2002)

J.D. Jentsch et al.

Impairments of reversal learning and response perseveration after repeated, intermittent cocaine administrations to monkeys

Neuropsychopharmacology

(2002)

B. Jones et al.

Limbic lesions and the problem of stimulus—reinforcement associations

Experimental Neurology

(1972)

P.W. Kodituwakku et al.

Emotion-related learning in individuals prenatally exposed to alcohol: An investigation of the relation between set shifting, extinction of responses, and behavior

Neuropsychologia

(2001)

M.P. Laakso et al.

Prefrontal volumes in habitually violent subjects with antisocial personality disorder and type 2 alcoholism

Psychiatry Research

(2002)

D. Lapierre et al.

Ventral frontal deficits in psychopathy: Neuropsychological test findings

Neuropsychologia

(1995)

S. Lough et al.

Measuring and modifying abnormal social cognition in frontal variant frontotemporal dementia

Journal of Psychosomatic Research

(2002)

P. Malloy et al.

The orbitomedial frontal syndrome

Archives of Clinical Neuropsychology

(1993)

M. Meunier et al.

Effects of orbital frontal and anterior cingulate lesions on object and spatial memory in rhesus monkeys

Neuropsychologia

(1997)

M.J. Millan et al.

Serotonin (5-HT)2C receptors tonically inhibit dopamine (DA) and noradrenaline (NA), but not 5-HT, release in the frontal cortex in vivo

Neuropharmacology

(1998)

M.Z. Mintzer et al.

Cognitive impairment in methadone maintenance patients

Drug and Alcohol Dependence

(2002)

D.G. Mitchell et al.

Risky decisions and response reversal: Is there evidence of orbitofrontal cortex dysfunction in psychopathic individuals?

Neuropsychologia

(2002)

M. Nomura

Effects of bifemelane on discrimination learning of serotonergic-dysfunction rats

Pharmacology, Biochemistry and Behavior

(1992)

T.J. Ornstein et al.

Profiles of cognitive dysfunction in chronic amphetamine and heroin abusers

Neuropsychopharmacology

(2000)

S.B. Park et al.

Tryptophan depletion in normal volunteers produces selective impairments in learning and memory

Neuropharmacology

(1994)

R.M. Ridley et al.

An analysis of visual object reversal learning in the marmoset after amphetamine and haloperidol

Pharmacology, Biochemistry and Behavior

(1981)

T.W. Robbins et al.

Limbic-striatal interactions in reward-related processes

Neuroscience and Biobehavioral Reviews

(1989)

R.D. Rogers et al.

Dissociable deficits in the decision-making cognition of chronic amphetamine abusers, opiate abusers, patients with focal damage to prefrontal cortex, and tryptophan-depleted normal volunteers: Evidence for monoaminergic mechanisms

Neuropsychopharmacology

(1999)

E.T. Rolls et al.

Responses of striatal neurons in the behaving monkey. 1. Head of the caudate nucleus

Behavioural Brain Research

(1983)

W. Schultz

Getting formal with dopamine and reward

Neuron

(2002)

A.G. Smith et al.

The dopamine D3/D2 receptor agonist 7-OH-DPAT induces cognitive impairment in the marmoset

Pharmacology, Biochemistry and Behavior

(1999)

G.E. Alexander et al.

Parallel organization of functionally segregated circuits linking basal ganglia and cortex

Annual Review of Neuroscience

(1986)

S.W. Anderson et al.

Impairment of social and moral behavior related to early damage in human prefrontal cortex

Nature Neuroscience

(1999)

A. Bechara et al.

Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making

Journal of Neuroscience

(1999)

A. Bechara et al.

Dissociation of working memory from decision making within the human prefrontal cortex

Journal of Neuroscience

(1998)

A. Bechara et al.

Characterization of the decision-making deficit of patients with ventromedial prefrontal cortex lesions

Brain

(2000)

A. Bechara et al.

Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex

Cerebral Cortex

(1996)

G.S. Berns et al.

Predictability modulates human brain response to reward

Journal of Neuroscience

(2001)

M. Best et al.

Evidence for a dysfunctional prefrontal circuit in patients with an impulsive aggressive disorder

Proceedings of the National Academy of Sciences USA

(2002)

J.C. Bizot et al.

Effects of imipramine-like drugs and serotonin uptake blockers on delay of reward in rats. Possible implication in the behavioral mechanism of action of antidepressants

Journal of Pharmacology and Experimental Therapeutics

(1988)

R.J. Blair et al.

Somatic markers and response reversal: Is there orbitofrontal cortex dysfunction in boys with psychopathic tendencies?

Journal of Abnormal Child Psychology

(2001)

R.N. Cardinal et al.

Impulsive choice induced in rats by lesions of the nucleus accumbens core

Science

(2001)

R.N. Cardinal et al.

The effects of d-amphetamine, chlordiazepoxide, alpha-flupenthixol and behavioural manipulations on choice of signalled and unsignalled delayed reinforcement in rats

Psychopharmacology

(2000)

D.R. Cherek et al.

Fenfluramine effects on impulsivity in a sample of adults with and without history of conduct disorder

Psychopharmacology

(2000)

Cited by (347)

The neural correlates of memory integration in value-based decision-making during human spatial navigation
2024, Neuropsychologia
In daily life, we often make decisions based on relative value of the options, and we often derive these values from segmenting or integrating the outcomes of past episodes in memory. The neural correlates involved in value-based decision-making have been extensively studied in the literature, but few studies have investigated this topic in decisions that require segmenting or integrating episodic memory from related sources, and even fewer studies examine it in the context of spatial navigation. Building on the computational models from our previous studies, the current study investigates the neural substrates involved in decisions that require people either segment or integrate wayfinding outcomes involving different goals, across virtual spatial navigation tasks with differing demands. We find that when decisions require computation of spatial distances for navigation options, but also evaluation of one's prior spatial navigation ability with the task, the estimated value of navigational choices (EV) modulates neural activity in the dorsomedial prefrontal (dmPFC) cortex and ventrolateral prefrontal (vlFPC) cortex. However, superior parietal cortex tracked EV when decision-making tasks only require spatial distance memory but not evaluation of spatial navigation ability. Our findings reveal divergent neural substrates of memory integration in value-based decision-making under different spatial processing demands.
Chronic corticosterone improves perseverative behavior in mice during sequential reversal learning
2023, Behavioural Brain Research
Stressful life events can both trigger development of psychiatric disorders and promote positive behavioral changes in response to adversities. The relationship between stress and cognitive flexibility is complex, and conflicting effects of stress manifest in both humans and laboratory animals.
To mirror the clinical situation where stressful life events impair mental health or promote behavioral change, we examined the post-exposure effects of stress on cognitive flexibility in mice.
We tested female C57BL/6JOlaHsd mice in the touchscreen-based sequential reversal learning test. Corticosterone (CORT) was used as a model of stress and was administered in the drinking water for two weeks before reversal learning. Control animals received drinking water without CORT. Behaviors in supplementary tests were included to exclude non-specific confounding effects of CORT and improve interpretation of the results.
CORT-treated mice were similar to controls on all touchscreen parameters before reversal. During the low accuracy phase of reversal learning, CORT reduced perseveration index, a measure of perseverative responding, but did not affect acquisition of the new reward contingency. This effect was not related to non-specific deficits in chamber activity. CORT increased anxiety-like behavior in the elevated zero maze test and repetitive digging in the marble burying test, reduced locomotor activity, but did not affect spontaneous alternation behavior.
CORT improved cognitive flexibility in the reversal learning test by extinguishing prepotent responses that were no longer rewarded, an effect possibly related to a stress-mediated increase in sensitivity to negative feedback that should be confirmed in a larger study.
Computational approaches to modeling gambling behaviour: Opportunities for understanding disordered gambling
2023, Neuroscience and Biobehavioral Reviews
Computational modeling has become an important tool in neuroscience and psychiatry research to provide insight into the cognitive processes underlying normal and pathological behavior. There are two modeling frameworks, reinforcement learning (RL) and drift diffusion modeling (DDM), that are well-developed in cognitive science, and have begun to be applied to Gambling Disorder. RL models focus on explaining how an agent uses reward to learn about the environment and make decisions based on outcomes. The DDM is a binary choice framework that breaks down decision making into psychologically meaningful components based on choice reaction time analyses. Both approaches have begun to yield insight into aspects of cognition that are important for, but not unique to, gambling, and thus relevant to the development of Gambling Disorder. However, these approaches also oversimplify or neglect various aspects of decision making seen in real-world gambling behavior. Gambling Disorder presents an opportunity for ‘bespoke’ modeling approaches to consider these neglected components. In this review, we discuss studies that have used RL and DDM frameworks to investigate some of the key cognitive components in gambling and Gambling Disorder. We also include an overview of Bayesian models, a methodology that could be useful for more tailored modeling approaches. We highlight areas in which computational modeling could enable progression in the investigation of the cognitive mechanisms relevant to gambling.
Inquiry of the orbitofrontal cortex role in incentive learning: An artificial neural networks simulation
2023, Cognitive Systems Research
This study evaluates, through artificial neural network (NN) simulations, the roles of amygdala (AMG) and orbitofrontal cortex (OFC) in reversal learning as responsible for assigning and updating incentive values, respectively, by taking into account some of its physiological characteristics and its connections with mesolimbic system structures. An architecture was built which contained modules of nodes representing OFC, amygdala (AMG), ventral tegmental area (VTA) and nucleus accumbens (NACC). Simulations were based on experimental responses of rats during a discrimination and reversal learning task in a T-maze. During simulations, intact NN responded similar to control rats. OFC showed retarded encoding in afferent connections as comparing with AMG nodes and an increase in the recruitment of neural nodes during reversal tasks. Activity of neural nodes plausibly emulates reinforcer preference during discriminations and reversals in AMG, COF and NACC. All these findings were disrupted by OFC’s inactivation. NN with OFC nodes inactivated (RN), just like rats with OFC inactivated, achieved discrimination learning although its reversal was impaired. As with rats, perseveration of incorrect responses was not found because subjects stop responding after a few incorrect trials. These findings are in accordance with the hypotheses of AMG as key in encoding novel predicting-outcome cues and OFC as critical for incentive value updating and the crucial role this prefrontal region plays for correct performance in discrimination tasks.
Reinforcement learning and Bayesian inference provide complementary models for the unique advantage of adolescents in stochastic reversal
2022, Developmental Cognitive Neuroscience
During adolescence, youth venture out, explore the wider world, and are challenged to learn how to navigate novel and uncertain environments. We investigated how performance changes across adolescent development in a stochastic, volatile reversal-learning task that uniquely taxes the balance of persistence and flexibility. In a sample of 291 participants aged 8–30, we found that in the mid-teen years, adolescents outperformed both younger and older participants. We developed two independent cognitive models, based on Reinforcement learning (RL) and Bayesian inference (BI). The RL parameter for learning from negative outcomes and the BI parameters specifying participants’ mental models were closest to optimal in mid-teen adolescents, suggesting a central role in adolescent cognitive processing. By contrast, persistence and noise parameters improved monotonically with age. We distilled the insights of RL and BI using principal component analysis and found that three shared components interacted to form the adolescent performance peak: adult-like behavioral quality, child-like time scales, and developmentally-unique processing of positive feedback. This research highlights adolescence as a neurodevelopmental window that can create performance advantages in volatile and uncertain environments. It also shows how detailed insights can be gleaned by using cognitive models in new ways.
An empirical evaluation of active inference in multi-armed bandits
2021, Neural Networks
A key feature of sequential decision making under uncertainty is a need to balance between exploiting—choosing the best action according to the current knowledge, and exploring—obtaining information about values of other actions. The multi-armed bandit problem, a classical task that captures this trade-off, served as a vehicle in machine learning for developing bandit algorithms that proved to be useful in numerous industrial applications. The active inference framework, an approach to sequential decision making recently developed in neuroscience for understanding human and animal behaviour, is distinguished by its sophisticated strategy for resolving the exploration–exploitation trade-off. This makes active inference an exciting alternative to already established bandit algorithms. Here we derive an efficient and scalable approximate active inference algorithm and compare it to two state-of-the-art bandit algorithms: Bayesian upper confidence bound and optimistic Thompson sampling. This comparison is done on two types of bandit problems: a stationary and a dynamic switching bandit. Our empirical evaluation shows that the active inference algorithm does not produce efficient long-term behaviour in stationary bandits. However, in the more challenging switching bandit problem active inference performs substantially better than the two state-of-the-art bandit algorithms. The results open exciting venues for further research in theoretical and applied machine learning, as well as lend additional credibility to active inference as a general framework for studying human and animal behaviour.

View all citing articles on Scopus

View full text

The neuropsychology of ventral prefrontal cortex: Decision-making and reversal learning

Abstract

Introduction

Section snippets

The cognitive neuroscience of decision-making

The cognitive neuroscience of reversal learning

The development of decision-making and reversal learning

Synthesis

Acknowledgments

Behavioral Brain Research

Trends in Cognitive Sciences

Neurobiology of Aging

Pharmacology and Biochemistry of Behavior

Neuropsychologia

Cognition

Physiology and Behavior

Neuropsychologia

Trends in Cognitive Sciences

Neuropsychologia

Neuropsychologia

Behavioral Brain Research

Cortex

Neuropsychopharmacology

Pharmacology, Biochemistry and Behavior

Neuropsychologia

Neuropsychopharmacology

Neuropsychopharmacology

Experimental Neurology

Neuropsychologia

Psychiatry Research

Neuropsychologia

Journal of Psychosomatic Research

Archives of Clinical Neuropsychology

Neuropsychologia

Neuropharmacology

Drug and Alcohol Dependence

Neuropsychologia

Pharmacology, Biochemistry and Behavior

Neuropsychopharmacology

Neuropharmacology

Pharmacology, Biochemistry and Behavior

Neuroscience and Biobehavioral Reviews

Neuropsychopharmacology

Behavioural Brain Research

Neuron

Pharmacology, Biochemistry and Behavior

Parallel organization of functionally segregated circuits linking basal ganglia and cortex

Annual Review of Neuroscience

Impairment of social and moral behavior related to early damage in human prefrontal cortex

Nature Neuroscience

Different contributions of the human amygdala and ventromedial prefrontal cortex to decision-making

Journal of Neuroscience

Dissociation of working memory from decision making within the human prefrontal cortex

Journal of Neuroscience

Characterization of the decision-making deficit of patients with ventromedial prefrontal cortex lesions

Brain

Failure to respond autonomically to anticipated future outcomes following damage to prefrontal cortex

Cerebral Cortex

Predictability modulates human brain response to reward

Journal of Neuroscience

Evidence for a dysfunctional prefrontal circuit in patients with an impulsive aggressive disorder

Proceedings of the National Academy of Sciences USA

Effects of imipramine-like drugs and serotonin uptake blockers on delay of reward in rats. Possible implication in the behavioral mechanism of action of antidepressants

Journal of Pharmacology and Experimental Therapeutics

Somatic markers and response reversal: Is there orbitofrontal cortex dysfunction in boys with psychopathic tendencies?

Journal of Abnormal Child Psychology

Impulsive choice induced in rats by lesions of the nucleus accumbens core

Science

The effects of d-amphetamine, chlordiazepoxide, alpha-flupenthixol and behavioural manipulations on choice of signalled and unsignalled delayed reinforcement in rats

Psychopharmacology

Fenfluramine effects on impulsivity in a sample of adults with and without history of conduct disorder

Psychopharmacology