1 / 33

Learning, Volatility and the ACC

Learning, Volatility and the ACC. Tim Behrens FMRIB + Psychology, University of Oxford FIL - UCL. i-1. 0.8. i-2. i-3. i-4. i-5. i-6. i-7. i-8. CON. 0.7. 0.6. 0.5. 0.4. 0.3. Reward History Weight (β). 0.2. 0.1. 0.0. -0.1. -0.2. Trials Into Past. B.

everly
Download Presentation

Learning, Volatility and the ACC

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Learning, Volatility and the ACC • Tim Behrens • FMRIB + Psychology, University of Oxford • FIL - UCL.

  2. i-1 0.8 i-2 i-3 i-4 i-5 i-6 i-7 i-8 CON 0.7 0.6 0.5 0.4 0.3 Reward History Weight (β) 0.2 0.1 0.0 -0.1 -0.2 Trials Into Past B Kennerley, et al., Nature Neuroscience, 2006

  3. i-1 0.8 i-2 i-3 i-4 i-5 i-6 i-7 i-8 CON 0.7 0.6 0.5 0.4 0.3 Reward History Weight (β) 0.2 0.1 0.0 -0.1 -0.2 Trials Into Past B ACCs Kennerley et al. Nature Neuroscience, 2006

  4. ACCG Monkeys will sacrifice food opportunities to look at other monkeys Rudebeck,et al. Science 2005

  5. ACCG Interest in other individuals is reduced after ACC gyrus lesion Rudebeck,et al. Science 2005

  6. Anatomy - Differences in connections between ACCs and ACCg. • Connections unique to the sulcus are mainly with motor regions: • Primary motor cortex • Premotor cortex • Parietal motor areas • Spinal Cord • ACCs has information about our own actions

  7. Anatomy - Differences in connections between ACCs and ACCg. • Connections unique to the gyrus are mainly with regions that process emotional and biological stimuli: • Periacqueductal grey • hypothalamus • STS/STG • Insula/Temporal pole connections are stronger to the gyrus • ACCg has access to information about other agents.

  8. Anatomy - shared connections between ACCs and ACCg. • Some shared connections • Orbitofrontal cortex • Amydala • Ventral striatum • ACCg and ACCs are strongly interconnected • Both regions have access to and influence over reward and value processing.

  9. ACC Sulcus and learning about your actions.

  10. i-1 0.8 i-2 i-3 i-4 i-5 i-6 i-7 i-8 CON 0.7 0.6 0.5 0.4 0.3 Reward History Weight (β) 0.2 0.1 0.0 -0.1 -0.2 Trials Into Past B ACCs Kennerley et al. Nature Neuroscience, 2006

  11. i-1 0.8 i-2 i-3 i-4 i-5 i-6 i-7 i-8 CON 0.7 0.6 0.5 0.4 Reward History Weight (β) 0.3 0.2 0.1 0.0 -0.1 -0.2 Trials Into Past What determines the integration length? Kennerly et al. Nat Neurosci 2006 Sugrue et al. Science 2005

  12. i-1 0.8 i-2 i-3 i-4 i-5 i-6 i-7 i-8 CON 0.7 0.6 0.5 0.4 Reward History Weight (β) 0.3 0.2 0.1 0.0 -0.1 -0.2 Trials Into Past VOLATILE Reward probabilities change approximately every 25 trials STABLE Reward probabilities change only after hundreds of trials Kennerly et al. Nat Neurosci 2006 Sugrue et al. Science 2005

  13. α x δ prediction (Vt) outcome new prediction (Vt+1) δ Reinforcement learning • We need to continually re-appraise the value of an action based each new experience.

  14. The learning rate is the weight given to the current information The prediction error is the information available from this event Updating beliefs on the basis of new information Vt+1=Vt +( α x δ ) 14

  15. The learning rate and the value of information. Vt+1=Vt +( α x δ ) The learning rate should represent the value of the current information for guiding future beliefs.

  16. α=0.01 α=0.1 α=0.4 Relationship with integration length

  17. stable 37 63 Behrens et al., Nature Neuroscience, 2007

  18. Vt+1=Vt+α x δ Behrens, Woolrich, Walton, Rushworth, Nature Neuroscience, 2007

  19. changes in reward estimates occur throughout the task… …as do change in volatility estimates Behrens, Woolrich, Walton, Rushworth, Nature Neuroscience, 2007

  20. Monitor x Volatility Decide Monitor Behrens et al., Nature Neuroscience, 2007

  21. ACC effect size predicts learning rate across subjects Behrens, Woolrich, Walton &Rushworth Nat Neurosci 2007

  22. ACC Gyrus and learning about your social partners.

  23. ACCG Interest in other individuals is reduced after ACC gyrus lesion Rudebeck et al. Science 2005

  24. Rudebeck et al., Science, 2006

  25. Learning about other agents 37 63 Behrens, Hunt, Woolrich, Rushworth Nature 2008

  26. Value of action information Value of social information Sources of information Probability that correct colour is blue Probability that confederate advice is good Behrens, Hunt, Woolrich, Rushworth Nature 2008

  27. Social information is integrated over time - behaviour

  28. Reward Prediction Error Vt+1=Vt +( α xδ ) Reward - Expectation Outcome Effect size Time Behrens, Hunt, Woolrich, Rushworth Nature 2008

  29. Prediction error on a social partner. Vt+1=Vt +( α xδ ) Lie event - Lie prediction Outcome Effect size Time Behrens, Hunt, Woolrich, Rushworth Nature 2008

  30. The value of information and the ACC Vt+1=Vt +( αx δ ) Value of reward information Value of social information 30

  31. Combining Information to drive behaviour Vt+1=Vt+( α x δ )

  32. Conclusions • ACC codes a learning signal when information is observed. • This signal predicts the speed of learning. • Learning from our own and others’ actions are processed in parallel in ACCs and ACCg. • The outputs of these parallel learning processes are combined in the reward system.

  33. Acknowledgments • Matthew Rushworth • Mark Woolrich • Laurence Hunt • Mark Walton 33

More Related