Willing to wait: Elevated reward-processing EEG activity associated with a greater preference for larger-but-delayed rewards

Highlights

• We tested the relationship between reward-processing and delay-discounting responses.

• Enhanced reward-anticipation and -outcome EEGs predicted larger-but-delayed choices.

• Reward-anticipation EEG refers to pre-feedback occipital alpha-suppression.

• Reward-outcome EEG refers to post-feedback frontal-midline theta and parietal beta.

• Hence, enhanced reward-processing was related to delay-gratification.

Abstract

While almost everyone discounts the value of future rewards over immediate rewards, people differ in their so-called delay-discounting. One of the several factors that may explain individual differences in delay-discounting is reward-processing. To study individual-differences in reward-processing, however, one needs to consider the heterogeneity of neural-activity at each reward-processing stage. Here using EEG, we separated reward-related neural activity into distinct reward-anticipation and reward-outcome stages using time-frequency characteristics. Thirty-seven individuals first completed a behavioral delay-discounting task. Then reward-processing EEG activity was assessed using a separate reward-learning task, called a reward time-estimation task. During this EEG task, participants were instructed to estimate time duration and were provided performance feedback on a trial-by-trial basis. Participants received monetary-reward for accurate-performance on Reward trials, but not on No-Reward trials. Reward trials, relative to No-Reward trials, enhanced EEG activity during both reward-anticipation (including, cued-locked delta power during cue-evaluation and pre-feedback alpha suppression during feedback-anticipation) and reward-outcome (including, feedback-locked delta, theta and beta power) stages. Moreover, all of these EEG indices correlated with behavioral performance in the time-estimation task, suggesting their essential roles in learning and adjusting performance to maximize winnings in a reward-learning situation. Importantly, enhanced EEG power during Reward trials, as reflected by stronger 1) pre-feedback alpha suppression, 2) feedback-locked theta and 3) feedback-locked beta, was associated with a greater preference for larger-but-delayed rewards in a separate, behavioral delay-discounting task. Results highlight the association between a stronger preference toward larger-but-delayed rewards and enhanced reward-processing. Moreover, our reward-processing EEG indices detail the specific stages of reward-processing where these associations occur.

Introduction

Choosing between receiving $400 today or $800 today is easy. Most people, if not everyone, will select $800 today. However, choosing between receiving $400 today or $800 in three years is more difficult, and different people will choose differently. This latter decision becomes harder and requires a stronger computational demand (Rangel et al., 2008) because the subjective value of $800 is devalued, or discounted, over time. For decades, economists, psychologists, and, more recently, cognitive neuroscientists have studied this so-called delay-discounting phenomenon (also known as temporal discounting or inter-temporal choices; Ainslie, 1975, Frederick et al., 2003, Kalenscher and Pennartz, 2008, Peters and Büchel, 2011, Samuelson, 1937, Schultz, 2010). While the phenomenon is well documented, it is clear that people vary in how much they discount future rewards. In fact, individual differences in delay-discounting are stable over time and are sometimes considered a personality trait (Kirby, 2009, Odum, 2011). Recently personality and cognitive-neuroscience research has shown that these individual differences in delay-discounting are correlated with several trait affective and cognitive variables (Civai et al., 2016, Hirsh et al., 2008, Mahalingam et al., 2014). Among these variables is reward-processing (Benningfield et al., 2014, Boettiger et al., 2007, Hariri et al., 2006), which relates to the value an individual places on potential rewards during both the expectation and receipt of that reward (McClure et al., 2004, Schultz et al., 2000). Yet, the exact nature of the relationship between delay-discounting tendencies and individual differences in reward-processing is still unclear, perhaps due to the multifaceted nature of reward-processing.

An early study by Hariri et al. (2006) reported that individuals with elevated reward-related neural activation in the ventral striatum (VS) during an incentivized fMRI card-guessing task had a stronger preference toward smaller-but-immediate rewards, as indexed by a subsequent behavioral delay-discounting task. This finding suggests that enhanced reward-processing is related to a stronger preference toward smaller-but-immediate rewards. However, recent data suggests the opposite pattern, indicating that elevated reward-processing is associated with a preference for larger-but-delayed (as opposed to smaller-but-immediate) rewards. In line with this view, a recent fMRI study using the Monetary Incentive Delay (MID) task reported that elevated VS activation during reward-anticipation among adolescents was associated with a stronger preference for larger-but-delayed rewards on a subsequent behavioral delay-discounting task (Benningfield et al., 2014). This finding is consistent with other recent fMRI studies reporting that elevated activation in the VS is associated with a stronger preference toward larger-but-delayed rewards (Ballard and Knutson, 2009, Samanez-Larkin et al., 2011). This relationship is also in line with animal research showing that lesions to the VS lead to a preference for smaller-but-immediate choices (Cardinal et al., 2001). In addition to the VS, enhanced activity in another neural region implicated in reward processing, the lateral orbitofrontal cortex (L-OFC), has been associated with a stronger preference toward larger-but-delayed rewards during an fMRI delay-discounting task (Boettiger et al., 2007). Next, there is indirect evidence from research involving the Val158Met polymorphism of the catechol-O-methyltransferase (COMT) gene. The Met-allele of the COMT gene is associated with higher synaptic dopamine levels and carriers of this allele display enhanced VS activation in an fMRI reward task (Chen et al., 2004, Yacubian et al., 2007). Critically, Met-allele carriers also show a preference toward larger-but-delayed rewards (Boettiger et al., 2007, Gianotti et al., 2012, Smith and Boettiger, 2012). More recently, research with Parkinson patients suggests that medications designed to elevate dopamine signaling are associated with a heightened preference for larger-but-delayed rewards (Foerde et al., 2016). Thus, taken as a whole, evidence from fMRI, animal, genetic, and pharmacological studies suggest that individuals with elevated reward-processing have a tendency to wait for larger rewards and forgo smaller-but-immediate rewards.¹

The current study aimed to further test and substantiate the relationship between a stronger preference toward larger-but-delayed rewards and enhanced reward-processing by investigating the relationship at different temporal stages of reward-processing via electroencephalogram (EEG). Reward-processing is thought be comprised of two temporal stages that are mediated by distinct neurobiological systems: reward-anticipation and reward-outcome (Berridge, 1996, Wise, 2008). The superior temporal resolution of EEG, compared to fMRI, allows researchers to more accurately dissociate neural-cognitive states that occur close to each other in time (Cohen, 2014, Luck, 2014), such as reward-anticipation and reward-outcome stages, as well as between different sub-stages within reward-anticipation itself (Brunia et al., 2011, Goldstein et al., 2006, McAdam and Seales, 1969). EEG, for instance, has been used to dissociate reward-anticipation from motor-preparation (Brunia et al., 2011, Hughes et al., 2013), which has been a challenge in previous fMRI research on the relationship between reward-processing and delay discounting tendencies. As noted by Benningfield et al. (2014), for example, the fMRI MID task does not isolate motor-preparation processes from reward-anticipation processes. Moreover, recent advancements in EEG time-frequency analyses allow researchers to investigate neural processes in ways that may not be available in other techniques, such as examining changes in neural activation (power) at a specific time windows and frequency bands (Cohen, 2014, Makeig et al., 2004).

To identify different stages of reward-processing, we adapted a feedback-learning task (called the reward time estimation task) (e.g., Damen and Brunia, 1987, Kotani et al., 2003, Luft, 2014, Pornpattananangkul and Nusslock, 2015). Here participants were asked to estimate a specific time duration by pressing a button 3.5 s after the onset of a Reward or No-Reward cue. The Reward/No-Reward cue indicated whether the current trial was a Reward- or No-Reward trial. Participants received monetary-reward for accurate-performance (pressing close to 3.5 s) on Reward trials, but not on No-Reward trials. Two seconds after making the button press, participants received feedback regarding the accuracy of their time-estimation. This paradigm allowed us to parse EEG related to reward-processing into reward-anticipation and reward-outcome phases (i.e., before and after feedback onset, respectively). Within reward-anticipation, we further separated EEG activity into 1) a cue-evaluation stage, involving a period immediately following the Reward/No-Reward cue, and 2) a feedback-anticipation stage, involving a period right before the feedback while participants were waiting to see if their recent action was considered accurate (i.e., close to 3.5 s). In addition to EEG indices, the improvement of time-estimation accuracy as the task proceeds can also be used as a behavioral index for the effectiveness in learning through feedback (Luft et al., 2013a). When the task provides rewards based on performance, then this behavioral index may reflect motivated-learning, or how motivated people are in learning to improve their performance in order to maximize reward earning (Luft, 2014). This motivated-learning behavioral index can be used 1) in corroborating EEG activity as indices of reward processing, and 2) in and of itself as an indirect, behavioral measure for individual-differences in reward-processing.

During cue-evaluation, participants evaluated whether their immediate future action could lead to a reward. In previous research, a cue signaling the possibility of receiving a reward was associated with a stronger P3 ERP component (compared to a cue in No-Reward trials) (Broyd et al., 2012, Goldstein et al., 2006, Ramsey and Finn, 1997, Santesso et al., 2012). More recently, Cavanagh (2015) found an enhancement of EEG power (or synchronization) in the delta band (~1–3 Hz, called cue-locked delta) to reward-related cues. This cue-locked delta power found predominantly at parietal sites approximately 100–500 ms following cue onset is relevant to the P3 ERP component. Moreover, Cavanagh (2015) showed that enhanced cue-locked delta power to reward-related cues predicted behavioral adjustments in a reinforcement learning task, suggesting an important role of cue-locked delta power in the cue evaluation stage.

As for feedback-anticipation, the suppression (or desynchronization) of alpha (~8–13 Hz) EEG power at parieto-occipital sites prior to stimulus onset appears to index anticipation-related processes toward upcoming visual stimuli. For instance, using a time-estimation task, Bastiaansen et al., 2002, Bastiaansen et al., 1999) and Bastiaansen and Brunia (2001) provided either visual or auditory feedback regarding the accuracy of time-estimation to participants. The authors reported alpha power suppression immediately preceding both types of feedback, but this alpha suppression was strongly distributed to occipital sites for the visual, but not auditory, feedback. This suggests the role of pre-stimulus alpha suppression in modality-specific anticipation processes. Given that alpha power reflects the functional inhibition of neural activity (Jensen and Mazaheri, 2010), alpha suppression during the pre-stimulus anticipatory period likely reflects the dis-inhibition of neural activity in sensory cortices to facilitate attentional allocation to the upcoming stimulus. Additionally, this pre-stimulus suppression of parieto-occipital alpha to visual stimuli has been found to track the location of where people anticipate the stimuli to appear (Thut et al., 2006). Stronger suppression of pre-stimulus alpha power is also associated with how well people perceive the preceding near-threshold stimulus (Hanslmayr et al., 2007). Collectively, these findings suggest a relationship between alpha-suppression and anticipatory attention. More recently, stronger alpha suppression has been reported following monetary-reward cues and preceding monetary-reward feedback (Hughes et al., 2013). This additional suppression of alpha power by reward motivational cues suggests that alpha suppression indexes enhanced attentional processes during the anticipation of reward-related stimuli/feedback. Consistent with this idea, van den Berg et al. (2014) recently investigated the role of reward-related cues in a Stroop task, and demonstrated an inter-individual relationship. Specifically, individuals who had particularly strong alpha suppression following reward-related cues were more likely to have better behavioral performance on reward trials. Therefore, pre-feedback alpha suppression should serve as a reliable index for individual differences in reward-processing during the feedback-anticipation stage of reward-anticipation.

The reward-outcome period in our time-estimation task involved participants receiving feedback for that particular trial. We focused on two types of feedback evaluation: reward-evaluation and performance-evaluation. During reward-evaluation, individuals evaluate the motivational value of the feedback (Luft, 2014). That is, regardless of whether their performance outcome is good or bad, people should be more motivated to learn the outcome of their performance when this performance can lead to monetary reward (i.e., during Reward trials compared to No-Reward trials). During performance-evaluation, individuals assess whether their prior action was good or bad in meeting their performance goal, e.g., whether or not their time-estimation was accurate (Cavanagh and Shackman, 2015, Miltner et al., 1997). The concept of performance evaluation has been related to both prediction error and conflict resolution. Unfortunately, many EEG studies have lumped these two aspects of feedback evaluation (i.e., reward-evaluation and performance evaluation) together, making it difficult to interpret the specific cognitive processes underlying their EEG findings. Here we separate reward-evaluation and performance evaluation. Furthermore, we examine three separate EEG indices during the reward-outcome stage, each occurring at distinct frequency bands: feedback-locked delta, feedback-locked theta and feedback-locked beta. As outlined next, we argue that each of these feedback-locked EEG profiles index individual differences in reward-processing during the reward-outcome stage.

First, similar to cue-locked delta, recent studies have started to document changes in feedback-locked delta (1–3 Hz) power at parietal sites during reward-outcome approximately 100–500 ms following feedback onset (Cavanagh, 2015, Foti et al., 2015, Leicht et al., 2013). Cavanagh (2015) reported that feedback-locked delta was associated with prediction error in a reinforcement-learning task. Foti et al. (2015) reported that feedback-locked delta activity was stronger to feedbacks indicating monetary gains, compared to losses. Nonetheless, because monetary-gain feedback in these previous studies indicated both good performance and a positive reward outcome, it remains unclear whether enhancement in feedback-locked delta activity to gain feedback is driven by performance-evaluation or reward-evaluation. In the present study, we use the time estimation task to dissociate neural processes associated with reward and performance evaluation, and we examine their respective relationships with individual differences in delay-discounting.

Next, the enhancement of feedback-locked theta (~4–7 Hz) power at frontal-midline sites (i.e., frontal-midline theta, FMT) approximately 200–400 ms following feedback onset has been implicated in feedback/outcome evaluation (Cohen et al., 2011). Thought to be generated from the anterior-cingulate cortex (Cavanagh and Frank, 2014), enhanced feedback-locked theta has been associated with cognitive-control processes that incorporate feedback/outcome information to facilitate behavioral adjustment on subsequent trials in order to maximize performance (van de Vijver et al., 2011). Feedback-locked theta appears sensitive to both performance-evaluation and reward-evaluation (Luft, 2014). In contrast to feedback-locked delta, feedback-locked theta is reliably stronger for bad-performance (compared to good-performance) feedback (Cohen et al., 2007; for a review see Luft (2014)). Additionally, such enhancement to bad-performance feedback predicts behavioral adjustment on a subsequent trial (Cavanagh and Shackman, 2015). As for reward-evaluation, Van den Berg et al. (2012) employed the time-estimation task and focused on the Feedback-Related Negativity (FRN), an event-related potential (ERP) thought to reflect the phase/time-locked feature of feedback-locked theta (Cavanagh et al., 2012). They found an enhanced FRN to feedback during Reward-trials where Good-performance led to monetary reward, compared to feedback during No-Reward trials where performance had no monetary consequences. Similar to the FRN, other studies have shown the influence of reward-evaluation on feedback-locked theta. For instance, feedback-locked theta is modulated by reward expectation (Cohen et al., 2007) and is stronger following feedback indicating a higher magnitude of monetary reward (Leicht et al., 2013). Thus, we expected feedback-locked theta to be modulated by both reward and performance evaluation.

Lastly, in addition to feedback-locked delta and theta, several recent studies have focused on feedback-locked EEG in the beta band (~15–25 Hz) (for review, see Luft (2014)). Similar to feedback-locked delta (but opposite to feedback-locked theta), researchers have consistently found stronger beta power to positive feedback (e.g., monetary gains), compared to negative feedback (e.g., monetary losses) (Cohen et al., 2007, De Pascalis et al., 2012, HajiHosseini et al., 2012, Marco-Pallares et al., 2008, Marco-Pallarés et al., 2009). Given that a similar pattern of enhanced beta power has been reported in the ventral striatum of animals during a reward-processing task (Berke, 2009, Courtemanche et al., 2003), it has been proposed that feedback-locked beta power represents reward-related signals from this region. Parietal feedback-locked beta power is reduced in humans following feedback during reward-learning tasks, such as the time estimation task (Luft et al., 2013a, Luft et al., 2013b, van de Vijver et al., 2011). This reduction in feedback-locked beta power (desynchronization) is less pronounced when feedback indicates good performance, compared to bad performance. Additionally, a greater enhancement of feedback-locked beta power (i.e., less reduction/desynchronization) predicts more rapid learning of a time duration in the time estimation task (Luft et al., 2013a). Nonetheless, similar to feedback-locked delta, most previous studies of feedback-locked beta have either lumped performance evaluation and reward evaluation feedback together (e.g., monetary gain indicating both good performance and reward associated with the performance) or focused solely on performance-evaluation, making it hard to interpret the psychological meaning of the effects. The current study separated the two aspects of feedback evaluation, and assessed their relationships with delay-discounting tendencies.

In the present study, participants completed a behavioral delay-discounting task and then a separate EEG reward time-estimation task.² Drawing on existing research (Benningfield et al., 2014, Boettiger et al., 2007, Foerde et al., 2016), we predict that enhanced reward-related neural activity will be associated with a greater preference for larger-but-delayed rewards. An important contribution of this study is that we examined the relationship between reward-related neural activity and delay discounting tendencies at different temporal stages of reward-processing based on EEG time-frequency characteristics. Within the reward-anticipation stage, elevated reward-processing was operationalized as 1) greater cued-locked delta activity during cue-evaluation and 2) greater pre-feedback alpha suppression during feedback-anticipation. Within the reward-outcome stage, elevated reward-processing was operationalized as greater feedback-locked delta, theta and beta activity. To help corroborate these EEG variables as indices of reward processing, we examined the relationship between EEG-related data and behavioral learning performance during the reward time estimation task, which reflects motivated learning (Luft et al., 2013a). We also expect this behavioral index for motivated learning to correlate with individual differences in delay-discounting in a manner that is similar to reward-processing EEG indices.