learning volatility and the acc n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Learning, Volatility and the ACC PowerPoint Presentation
Download Presentation
Learning, Volatility and the ACC

Loading in 2 Seconds...

play fullscreen
1 / 33

Learning, Volatility and the ACC - PowerPoint PPT Presentation


  • 51 Views
  • Uploaded on

Learning, Volatility and the ACC. Tim Behrens FMRIB + Psychology, University of Oxford FIL - UCL. i-1. 0.8. i-2. i-3. i-4. i-5. i-6. i-7. i-8. CON. 0.7. 0.6. 0.5. 0.4. 0.3. Reward History Weight (β). 0.2. 0.1. 0.0. -0.1. -0.2. Trials Into Past. B.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Learning, Volatility and the ACC' - everly


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
learning volatility and the acc
Learning, Volatility and the ACC
  • Tim Behrens
  • FMRIB + Psychology, University of Oxford
  • FIL - UCL.
slide2

i-1

0.8

i-2

i-3

i-4

i-5

i-6

i-7

i-8

CON

0.7

0.6

0.5

0.4

0.3

Reward History Weight (β)

0.2

0.1

0.0

-0.1

-0.2

Trials Into Past

B

Kennerley, et al., Nature

Neuroscience, 2006

slide3

i-1

0.8

i-2

i-3

i-4

i-5

i-6

i-7

i-8

CON

0.7

0.6

0.5

0.4

0.3

Reward History Weight (β)

0.2

0.1

0.0

-0.1

-0.2

Trials Into Past

B

ACCs

Kennerley et al. Nature

Neuroscience, 2006

slide4

ACCG

Monkeys will sacrifice food

opportunities to look at other monkeys

Rudebeck,et al. Science 2005

slide5

ACCG

Interest in other individuals

is reduced after ACC gyrus lesion

Rudebeck,et al. Science 2005

anatomy differences in connections between accs and accg
Anatomy - Differences in connections between ACCs and ACCg.
  • Connections unique to the sulcus are mainly with motor regions:
    • Primary motor cortex
    • Premotor cortex
    • Parietal motor areas
    • Spinal Cord
  • ACCs has information about our own actions
anatomy differences in connections between accs and accg1
Anatomy - Differences in connections between ACCs and ACCg.
  • Connections unique to the gyrus are mainly with regions that process emotional and biological stimuli:
    • Periacqueductal grey
    • hypothalamus
    • STS/STG
    • Insula/Temporal pole connections are stronger to the gyrus
  • ACCg has access to information about other agents.
anatomy shared connections between accs and accg
Anatomy - shared connections between ACCs and ACCg.
  • Some shared connections
    • Orbitofrontal cortex
    • Amydala
    • Ventral striatum
    • ACCg and ACCs are strongly interconnected
  • Both regions have access to and influence over reward and value processing.
slide10

i-1

0.8

i-2

i-3

i-4

i-5

i-6

i-7

i-8

CON

0.7

0.6

0.5

0.4

0.3

Reward History Weight (β)

0.2

0.1

0.0

-0.1

-0.2

Trials Into Past

B

ACCs

Kennerley et al. Nature

Neuroscience, 2006

slide11

i-1

0.8

i-2

i-3

i-4

i-5

i-6

i-7

i-8

CON

0.7

0.6

0.5

0.4

Reward History Weight (β)

0.3

0.2

0.1

0.0

-0.1

-0.2

Trials Into Past

What determines the integration length?

Kennerly et al. Nat Neurosci 2006

Sugrue et al. Science 2005

slide12

i-1

0.8

i-2

i-3

i-4

i-5

i-6

i-7

i-8

CON

0.7

0.6

0.5

0.4

Reward History Weight (β)

0.3

0.2

0.1

0.0

-0.1

-0.2

Trials Into Past

VOLATILE

Reward probabilities change

approximately every 25 trials

STABLE

Reward probabilities change

only after hundreds of trials

Kennerly et al. Nat Neurosci 2006

Sugrue et al. Science 2005

reinforcement learning

α x δ

prediction

(Vt)

outcome

new prediction

(Vt+1)

δ

Reinforcement learning
  • We need to continually re-appraise the value of an action based each new experience.
updating beliefs on the basis of new information

The learning rate is the

weight given to the

current information

The prediction error

is the information

available from this event

Updating beliefs on the basis of new information

Vt+1=Vt +( α x δ )

14

the learning rate and the value of information
The learning rate and the value of information.

Vt+1=Vt +( α x δ )

The learning rate should represent the

value of the current information

for guiding future beliefs.

slide17

stable

37

63

Behrens et al., Nature Neuroscience, 2007

slide18

Vt+1=Vt+α x δ

Behrens, Woolrich, Walton, Rushworth, Nature Neuroscience, 2007

changes in reward estimates occur throughout the task
changes in reward estimates occur throughout the task…

…as do change in volatility estimates

Behrens, Woolrich, Walton, Rushworth, Nature Neuroscience, 2007

slide20

Monitor

x

Volatility

Decide

Monitor

Behrens et al., Nature Neuroscience, 2007

acc effect size predicts learning rate across subjects
ACC effect size predicts learning rate across subjects

Behrens, Woolrich, Walton &Rushworth Nat Neurosci 2007

slide23

ACCG

Interest in other individuals

is reduced after ACC gyrus lesion

Rudebeck et al. Science 2005

learning about other agents
Learning about other agents

37

63

Behrens, Hunt, Woolrich, Rushworth Nature 2008

sources of information

Value of action information

Value of social information

Sources of information

Probability that correct colour is blue

Probability that confederate advice is good

Behrens, Hunt, Woolrich, Rushworth Nature 2008

reward prediction error
Reward Prediction Error

Vt+1=Vt +( α xδ )

Reward -

Expectation

Outcome

Effect size

Time

Behrens, Hunt, Woolrich, Rushworth Nature 2008

prediction error on a social partner
Prediction error on a social partner.

Vt+1=Vt +( α xδ )

Lie event -

Lie prediction

Outcome

Effect size

Time

Behrens, Hunt, Woolrich, Rushworth Nature 2008

the value of information and the acc
The value of information and the ACC

Vt+1=Vt +( αx δ )

Value of reward information

Value of social information

30

conclusions
Conclusions
  • ACC codes a learning signal when information is observed.
  • This signal predicts the speed of learning.
  • Learning from our own and others’ actions are processed in parallel in ACCs and ACCg.
  • The outputs of these parallel learning processes are combined in the reward system.
acknowledgments
Acknowledgments
  • Matthew Rushworth
  • Mark Woolrich
  • Laurence Hunt
  • Mark Walton

33