1 / 48

Reward-related Neural Circuitry

Reward-related Neural Circuitry. Julie Fiez, Ph.D. Departments of Psychology & Neuroscience. Acknowledgements. Karin Cox Mauricio Delgado Corrine Durisko Mary Conway Kate Fissell Chris May Alison Moed Susan Ravizza Elizabeth Tricomi Steve Wilson. Bruce McCandliss James McClelland

prisca
Download Presentation

Reward-related Neural Circuitry

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reward-related Neural Circuitry Julie Fiez, Ph.D. Departments of Psychology & Neuroscience

  2. Acknowledgements Karin Cox Mauricio Delgado Corrine Durisko Mary Conway Kate Fissell Chris May Alison Moed Susan Ravizza Elizabeth Tricomi Steve Wilson Bruce McCandliss James McClelland Athanassio Protopapas Michael Sayette Andy Stegner

  3. Dopamine Plays a Crucial Role in Reward-Related Processing Dopamine neurons respond to unexpected rewards. Schultz et al. (1997). Science, 275:1593-1599 Animals will work for delivery of drugs that stimulate dopaminergic signalling.

  4. Dopamine neurons project into distinct fronto-striatal-thalamic loops PFC Orbitofrontal THALAMUS Dorsal Striatum (Caudate/Putamen) Ventral Striatum (Nucleus Accumbens) SNpc VTA

  5. Is Dopamine a “Pleasure” Signal?“Liking” vs. “Wanting” Cannon & Bseirki (2004). Physiol & Behav, 81:741-7428.

  6. Does Dopamine Support the Development of Associations That Yield Increased Reward? Even simple behaviors have multiple opportunities for “habit” formation: light -> lever press -> food delivery stim -> response -> outcome Stimulus-outcome: consequences (feedback) may alter the value of neutral stimulus Response-outcome: consequences may alter motor (and cognitive) activity Stimulus-response-outcome: consequences may alter the relationship between a stimulus & a response Stimulus-response: after learning, behavior may be no longer governed by outcomes

  7. The Dopamine Signal May be Ideal to Support Such Reinforcement Learning Egelman et al. (1998). J Cogn Neurosci, 10:623-30. Schultz & Montague(1997). Science, 275:1593-1599

  8. Do ventral & dorsal striatum support different aspects of reinforcement learning? (e.g., Elliott et al., 2004; O’Doherty et al., 2004; Robbins et al., 1992) • Training: • initial Pavlovian training • CS+: light paired with drug delivery • CS-: clicks presented non-contingently • 2nd order conditioning • each lever press leads to light (CS+) delivery • 10 lever presses earns drug delivery • drug delivered after a fixed (20 min) interval Ito et al. (2002). J Neurosci, 22:6247-6253

  9. Emerging Issues for fMRI • What striatal response properties are observed in humans? • Are there dissociations between ventral vs. dorsal activity that converge with the animal literature? • What insight might such dissociations provide into the nature of human reward-related processing?

  10. Do striatal regions respond to the unpredictable delivery of reinforcers? Yes, especially at or near the nucleus accumbens: Schultz et al. (1997). Science, 275:1593-1599 Berns et al. (2001). J Neurosci, 21:1793-2798

  11. Do striatal regions respond to delivery of unexpected monetary outcomes? No significant differences between reward, punishment, and neutral trials were observed.

  12. How might we reconcile these findings? • The study by Berns & colleagues involved the delivery of a primary reinforcer. • The oddball study made use of an abstract, unconditioned cue (red or green arrow) to indicate gain or loss of a secondary reinforcer (delivered later). Schultz et al. (1997). Science, 275:1593-1599 • Will delivery of an unexpected, conditioned cue activate the ventral striatum?

  13. Unexpected delivery of conditioned cues • Male heavy smokers (at least 20 cigarettes/day) • Participants abstained from smoking for 8 hours • Compliance assessed by expired CO • Three neutral and one conditioned cue exposure Notepad Notepad Golf ball Golf ball Run 1 Run 1 … … Runs separated by approximately 23 minutes Runs separated by approximately 23 minutes Tape (neutral) Tape (neutral) Cigarette Cigarette Run 2 Run 2

  14. Interim Summary Consistent with prior neurophysiological findings, the ventral striatum responds to the unexpected delivery of primary reinforcers and conditioned cues. These findings support claims that the ventral striatum plays an integral role in reward-related signaling under normal conditions, and that it may contribute to pathological states such as addiction.

  15. PFC Orbitofrontal THALAMUS Dorsal Striatum Ventral Striatum SNpc VTA What about the dorsal striatum? Reward-responsive dopamine neurons also project to the dorsal striatum. The dorsal striatum has typically been observed to respond weakly in paradigms that drive the ventral striatum. However, robust reward-related differences have been found in the dorsal striatum using other paradigms.

  16. The Card Guessing Task Indicated monetary gain ? 7 Indicated monetary loss Trial Events: Outcome Card Card REWARD TRIAL Scanning Sequence: Scan 1 Scan 2 Scan 3 Scan 4 Scan 5 Scan 1 Choice Period Post-Outcome Period TEMPORAL SEQUENCE 0 3 6 9 12 15 Seconds

  17. Robust dorsal striatal activity is found during the card guessing task

  18. Which aspects of the task account for activation? Oddball task Guessing task 7 ? 7 ? • Unlike the ventral striatum, delivery of reinforcer or conditioned cue is not sufficient to activate dorsal striatum. • Activation during guessing task shows such delivery is not necessary. • Is it the mere need for an instrumental response? Blue circle = single keypress • Or must there be a real or perceived contingency between the the response & the outcome? Yellow circle = choose a keypress

  19. No choice trial Choice trial The dorsal striatum is sensitive to perceived response-outcome contingency.

  20. Involvement in response-outcome signaling may apply to complex situations. Caudate Activity LateTrials Early Trials

  21. 100 80 % heard as “lake” 60 40 Native Japanese Speaker 20 Native English Speakers 0 Natural Natural equal intermediate levels "rake" token "lake" token Speech Token Do the contributions of the dorsal striatum extend to “cold” cognition?

  22. The Development of Speech Categories May Be Self-Organizing When one neuron A participates in firing another neuron B, the strength of the effect of A on the firing of B is increased. - paraphrased from Hebb, 1949 Or, put more simply: Neurons that wire together, fire together.

  23. Once perceptual categories have been formed, can they be “reshaped”? Difficulties caused by a self-reinforcing tendency to hear two speech sounds as the same, thus: • Exaggerating the differences between sounds could overcome barrier. • Learning should not require explicit feedback.

  24. Fixed Training Condition 100 90 80 70 60 50 40 30 20 10 0 0.0 0.5 1.0 An Empirical Test of the Theory Adaptive Training Condition 100 90 80 Pretest 70 60 Posttest 50 40 30 20 10 0 0.0 0.5 1.0 [l] [r] [l] [r] Anchor Anchor Anchor Anchor

  25. Is the model complete? Difficulties caused by a self-reinforcing tendency to hear two speech sounds as the same, thus: • Exaggerating the differences between sounds could overcome barrier. • Learning should not require explicit feedback. • But what if feedback is given?

  26. Effects of Training Without Feedback Effects of Training With Feedback With feedback, both the adaptive and fixed techniques are effective. (McCandliss et al., 2002)

  27. Could the differences in learning reflect the engagement of the dorsal striatum? • Hypothesis: • In a motivated learner, performance feedback may be rewarding (correct response) or non-rewarding (incorrect response). • Outcomes may engage striatal reinforcement learning mechanisms. • Perceptual representations and associated responses that lead to “rewarding” outcomes are strengthened. • Test by having Japanese subjects perform the /r/ vs. /l/ task with and without feedback. • Compare activation in perceptual identification task to activation in the guessing task.

  28. A comparison across tasks. Guessing Task Categorizaton Task

  29. Increased Caudate Activation During Feedback Training The striatum is more active in the feedback as compared to the no-feedback condition.

  30. Performance Feedback Acts Like Gambling Reward/Punishment The activation is similar in location and pattern to that observed with the guessing task.

  31. Temporal cortex may be affected by top-down outcome signals.

  32. d d d d d d d d d d a a a a a a a a a a o o o o o o o o o o r r r l l l l l l l t i m e b i n s 1 - 5 t i m e b i n s 1 - 5 a f t e r o d d b a l l o n s e t t i m e b i n s 1 - 5 a f t e r o d d b a l l o n s e t a f t e r o d d b a l l o n s e t Can we see pre vs. post training differences? No explicit task: Subjects listen passively to stimuli An “oddball” response is presented every 16-24 ms Use fMRI to determine which areas of the brain respond to the oddball stimulus. If the sounds are perceived as the same, there should be no response to the oddballs.

  33. Examine the Neural Response to Native vs. Non-native Phoneme Contrast • Subjects: native Japanese speakers (n=9)

  34. Before training, auditory regions responded most to the native oddballs.

  35. After training, the largest responses were to the non-native oddballs.

  36. Implications for Perceptual Organization • The organization of perceptual categories may be mediated by both Hebbian-based and reinforcement-based learning mechanism. • During development, both mechanisms may come into play. Rewarding outcome: Adaptive input: Proportion of canonical syllables Baseline Social response Extinction Test periods (10 min) Kuhl, Nature Neuroscience Reviews, 5:831-843. Goldstein et al., PNAS, 100:830-835.

  37. Feedback may invoke learning that cuts across both implicit & explicit memory tasks.

  38. Implications for Normal Development The striatum appears to be part of a reinforcement learning system. This system may use rewarding outcomes (broadly construed) to shape: - perceptual representations of environmental stimuli - affective (motivational) responses evoked by stimuli & associated contexts - overt (motor) & covert (?) responses elicited by stimuli - episodic memory associations or retrieval processes

  39. Dysfunction/abnormal input into this system may result in developmental disorders. - susceptibility to drug abuse and drug addiction: - OCD - stress during early developmental periods

  40. Conclusions Ventral striatum is responsive to the mere presentation of primary reinforcers and conditioned cues; thus, the ventral striatum may play an important role in representing the incentive value of stimuli. Dorsal striatum is sensitive to whether there is a perceived contingency between a response and an outcome; thus, dorsal striatum may contribute to selecting and shaping behavior by associating actions with their outcomes. The dorsal striatum and prefrontal cortex may work together to provide substantial cognitive control over representations of incentive value induced by stimulus events.

  41. The dorsal striatal response is multi-faceted. The outcome period shows a sensitivity to outcome value: The choice period shows a sensitivity to motivational state: Large Reward Trial e 7 7 7 7 ? ? ? ? Positive Feedback $4.00 Positive Feedback $4.00 Positive Feedback $4.00 Positive Feedback $4.00 Low reward trial ? ? ? ? 3 3 3 3 Positive Feedback $0.00 Positive Feedback $0.00 Positive Feedback $0.00 Positive Feedback $0.00 Cue Cue Cue Cue Choice Choice Choice Choice - - - - Period Period Period Period Outcome Outcome Outcome Outcome Feedback Feedback Feedback Feedback High or Low High or Low

  42. rewarded movement instruction trigger reward unrewarded movement instruction trigger sound Caudate neurons show selective activation for trials in which the monkey’s movement will be rewarded (Schultz, Tremblay, and Hollerman, 2000)

  43. Modulation of cue-induced craving All participants refrained from smoking for 8 hours 10 participants expected to smoke midway through scanning session 10 participants did not expect to smoke Notepad Notepad Golf ball Golf ball Run 1 Run 1 … … Runs separated by approximately 23 minutes Runs separated by approximately 23 minutes Tape (neutral) Tape (neutral) Cigarette Cigarette Run 2 Run 2 Expectancy modulates the cue-induced response: • affects measures self-reported craving • affects facial expressions evoked in response to a conditioned cue • affects performance on tasks requiring executive control

  44. PFC Orbitofrontal THALAMUS Dorsal Striatum Ventral Striatum SNpc VTA The dorsal striatum may act in concert with prefrontal regions. Leon & Shadlen (1999). Neuron, 24:415-425.

  45. Right Ventrolateral PFC Left Ventrolateral PFC 6 6 5 5 4 4 3 3 Percent change from neutral 2 2 Percent change from neutral 1 1 0 0 -1 -1 -2 -2 -3 -3 NO YES NO YES Right Dorsolateral PFC Left Dorsolateral PFC 0.1 Percent change from neutral 1 1 1 Percent change from neutral NO YES NO YES Expectancy modulates prefrontal activity Dorsolateral PFC Ventrolateral PFC

  46. The dorsal striatum is sensitive to perceived response-outcome contingency.

  47. Behavioral Results After the imaging study, subjects completed extended training. With presentation of fixed (non-adpative stimuil), robust learning occurred only with feedback.

More Related