1 / 30

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence. The Aims of the Course: We will use the challenge of understanding the mechanisms of visuomotor coordination and action recognition in the monkey brain to provide a structured set of goals for our mastery of

porter-wright
Download Presentation

Michael Arbib: CS564 - Brain Theory and Artificial Intelligence

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Michael Arbib: CS564 - Brain Theory and Artificial Intelligence • The Aims of the Course: • We will use the challenge of understanding the mechanisms of visuomotor coordination and action recognition in the monkey brain to provide a structured set of goals for our mastery of • Brain Theory: modeling interactions of components of the brain, especially more or less realistic biological neural networks localized in specific brain regions • and • Connectionism in both Artificial intelligence (AI) and Cognitive psychology: modeling artificial neural networks -- networks of trainable “quasi-neurons” -- to provide “parallel distributed models” of intelligence in humans, animals and machines • Tools: Gaining an understanding of the use of NSLJ (Neural Simulation Language in Java) for simulating and analyzing neural networks

  2. CS 564: Brain Theory and Artificial Intelligence • URL:http://www-scf.usc.edu/~csci564/ for syllabus, instructor and TA information, handouts, homework and grades • DEN URL: http://den.usc.edu/academic/fa2001/courses/csci564/csci564.asp • Instructor: • Michael Arbib; arbib@pollux.usc.edu (Office hours: 11-12 Tuesdays, HNB 03) • TAs: Erhan Oztop, erhan@java.usc.edu, Salvador Marmol, smarmol@rana • This course provides a basic understanding of brain function, of artificial neural networks which provide tools for a new paradigm for adaptive parallel computation, and of the Neural Simulation Language NSLJ which allows us to simulate biological and artificial neural networks • No background in neuroscience is required, nor is specific programming expertise, but knowledge of Java will enable students to extend the NSLJ functionality in interesting ways

  3. Texts • Texts: MA Arbib, 1989, The Metaphorical Brain 2: Neural Networks and Beyond, Wiley-Interscience • TMB2 is being sent over to "The paper clip" for duplication. Students can purchase it there for $16 plus tax.A Weitzenfeld, MA Arbib and A Alexander, 2002, NSL Neural Simulation Language, MIT Press (in press)[ • An old version is at http://www-hbp.usc.edu/_Documentation/NSL/ Book/TOC.htm. A new version will be posted in a week or two.] • Other Required Reading: • will be posted on the Website – starting with the Mirror Neuron Proposal • Supplementary reading: • MA Arbib, Ed, 1995, The Handbook of Brain Theory and Neural Networks, MIT Press (paperback) • Michael A Arbib, and Jeffrey Grethe, Editors, 2001, Computing the Brain: A Guide to Neuroinformatics, San Diego: Academic Press (in press)

  4. Grading • One mid-term and a final will cover the entire contents of the readings as well as the lectures • Students will be organized into 5 groups, each working together on a semester-long project • The final exam will cover all of the course, but emphasizing material not covered in the mid-term • Distribution of Grades: • NSL assignments and other homework: 25%; • Mid-term: 20%; • Project 30%; • Final Exam: 25%

  5. 5 Specific Aims for the Mirror Neuron Proposal • 5 Specific Group Projects for CS 564 • Development of the Mirror System: • 1.Development of Grasp Specificity in F5 Motor and Canonical Neurons • 2. Visual Feedback for Grasping: A Possible Precursor of the Mirror Property • Recognition of Novel and Compound Actions and their Context: • 3.The Pliers Experiment: Extending the Visual Vocabulary • 4. Recognition of Compounds of Known Movements • 5. From Action Recognition to Understanding: Context and Expectation

  6. Send email to arbib@pollux.usc.eduby Noon on Tuesday September 4, 2001 • Line 1: • Your name then a colon (:) then your email address • Line 2: • Your Department (and your location if you are off-campus) • Line 3: • Any special skills you bring to the course • Lines 4+: • Your top 2 or 3 choices for a Project topic from the list: • 1.Development of Grasp Specificity • 2. Visual Feedback for Grasping • 3.The Pliers Experiment • 4. Recognition of Compound Movements • 5. Context and Expectation

  7. Michael Arbib:CS564 - Brain Theory and Artificial Intelligence • Lecture 1 Introduction and Brain Overview • Reading Assignments: • TMB2: Chapter 1 • Mirror Neuron Proposal* • * Read this “lightly” the first week of class We will master it and much more as the class proceeds Your initial task is to review the 5 specific aims, and pick the 2 you find most interesting One of these will probably become your project goal for the semester

  8. Opposition Spaces and Virtual Fingers The goal of a successful preshape, reach and grasp is to match the opposition axis defined by the virtual fingers of the hand with the opposition axis defined by an affordance of the object (Iberall and Arbib 1990)

  9. Visual Control of Grasping in Macaque Monkey A key theme of visuomotor coordination: parietal affordances (AIP) drive frontal motor schemas (F5) AIP - grasp affordances in parietal cortex Hideo Sakata F5 - grasp commands in premotor cortex Giacomo Rizzolatti

  10. Grasp Specificity in an F5 Neuron • Precision pinch (top) • Power grasp (bottom) • (Data from Rizzolatti et al.)

  11. AIP Dorsal Stream: Affordances Ways to grab this “thing” Task Constraints (F6) Working Memory (46?) Instruction Stimuli (F2) F5 Ventral Stream: Recognition “It’s a mug” IT PFC FARS (Fagg-Arbib-Rizzolatti-Sakata)Model Overview AIP extracts affordances - features of the object relevant to physical interaction with it. Prefrontal cortex provides “context” so F5 may select an appropriate affordance

  12. Syllabus Overview 1 • Introduction [Proposal] {Background: TMB Chapter 1} • Charting the Brain 1 [TMB 2.4] • The Brain as a Network of Neurons [TMB Section 2.3] • Visual Preprocessing [TMB 3.3] • Adaptive networks: Hebbian learning, Perceptrons; Landmark learning [TMB 3.4] [NSLbook] • Hebbian Learning and Visual plasticity; Self-organizing feature maps; [NSLJ] Kohonen maps • Higher level vision 1: object recognition {Background TMB 5.2} • Introduction to NSL: modules; SCS schematic capture system; Maxselector model[NSLbook] {Homework}

  13. Syllabus Overview 2 • Schemas for Reaching and Grasping; Affordances [TMB 2.2, 5.3] {Background TMB 2.1, 52} • Charting the Brain 2 • The FARS model of control of grasping 1: Population coding • The FARS model of control of grasping 2: Sequence learning and the basal ganglia • The FARS model of control of grasping 3: Working Memory

  14. Mirror Neurons Rizzolatti, Fadiga, Gallese, and Fogassi, 1995:Premotor cortex and the recognition of motor actions Mirror neurons form the subset of grasp-related premotor neurons of F5 which discharge when the monkey observes meaningful hand movements made by the experimenter or another monkey. F5 is endowed with anobservation/execution matching system [The non-mirror grasp neurons of F5 are called F5 canonical neurons.]

  15. Computing the Mirror System Response • The FARS Model: • Recognize object affordances and determine appropriate grasp. • The Mirror Neuron System (MNS) Model: • We must add recognition of • trajectoryand • hand preshape • to • recognition of object affordances • and ensure that all three are congruent. • There are parietal systems other than AIP adapted to this task.

  16. Object features cIPS Object affordance extraction F5canonical AIP Object affordance PF Motor -hand state program Cortex Integrate association (Grasp) temporal association Hand Visual shape Action recognition Motor Mirror recognition execution Feedback Motor Hand (Mirror program M1 motion Neurons) Hand-Object (Reach) detection spatial relation F5mirror F4 analysis STS work with Erhan Oztop 7a Object location The Mirror Neuron System (MNS) Model

  17. STS hand shape recognition Model Matching Precision grasp Hand Configuration Classification Color Coded Hand Feature Extraction Step 1 of hand shape recognition: system processes the color-coded hand image and generates a set of features to be used by the second step Step 2: The feature vector generated by the first step is used to fit a 3D-kinematics model of the hand by the model matching module. The resulting hand configuration is sent to the classification module.

  18. Virtual Hand/Arm and Reach/Grasp Simulator A precision pinch A power grasp and a side grasp

  19. Object Affordances Motor program Object affordance -hand state association Integrate temporal association F5canonical Hand shape recognition & Hand motion detection Action recognition (Mirror Neurons) Motor program Mirror Feedback Motor execution Hand-Object spatial relation analysis F5mirror Core Mirror Circuit Object affordance Mirror Neurons (F5mirror) Association (7b) Neurons Mirror Neuron Output Hand state Motor Program (F5 canonical) Mirror Feedback

  20. (a) Precision Pinch Mirror Neuron (b) Power Grasp Mirror Neuron Power and precision grasp resolution

  21. Syllabus Overview 3 • Reinforcement learning and motor control; [NSLJ] Conditional motor learning • Adaptive networks: Gradient descent and backpropagation [TMB 82] • [NSLJ] Backprop: a How to run the model; b How to write the model [NSLbook] • NeuroBench and the NeuroHomology Database

  22. Syllabus Overview 4 • The MNS1 Model 1: Hand Recognition • Systems concepts; Feedback and the spinal cord [TMB 31, 32] • The MNS 1 Model 2: Simulating the kinematics and biomechanics of reach and grasp • The MNS1 Model 3: Modeling the Core Mirror Neuron Circuit

  23. 5 Specific Aims for the Mirror Neuron Proposal • 5 Specific Group Projects for CS 564 • Development of the Mirror System: • 1.Development of Grasp Specificity in F5 Motor and Canonical Neurons • 2. Visual Feedback for Grasping: A Possible Precursor of the Mirror Property • Recognition of Novel and Compound Actions and their Context: • 3.The Pliers Experiment: Extending the Visual Vocabulary • 4. Recognition of Compounds of Known Movements • 5. From Action Recognition to Understanding: Context and Expectation

  24. A Human Mirror System • Rizzolatti, Fadiga, Matelli, Bettinardi, Perani, and Fazio:Broca's region is activated by observation of hand gestures: a PET study. • PET study of human brain with 3 experimental conditions: • Object observation (control condition) • Grasping observation • Object prehension. • The most striking result was highly significant activation in the rostral part of Broca's area. A key language area!!!

  25. A New Approach to the Evolution of Human Language • Rizzolatti, G, and Arbib, M.A., 1998, Language Within Our Grasp, Trends in Neuroscience, 21(5):188-194: • The Mirror System Hypothesis: Human Broca’s areacontains a mirror system for grasping which is homologous to the F5 mirror system of monkey, and this provides the evolutionary basis for language parity - i.e., an utterance means roughly the same for both speaker and hearer. • This adds a neural “missing link” to the tradition that roots speech in a prior system for communication based on manual gesture. • Beyond the Mirror: Seeing F5 as part of a larger mirror system, then extending our understanding via imitation to language-readiness. • This topic will be the jumping off point for the Spring 2002 version of CS 664, which will be taught by Professors Arbib and Itti.

  26. Syllabus Overview 5 • Control of eye movements [TMB 6.2] • Basal Ganglia and Control of eye movements - Dominey [NSLbook] • Dopamine and Sequence Learning • Abstract models of Sequence Learning • Extending the FARS model to mirror neurons and language • Project Reports 1, 2,3 • Project Reports 4,5; Concluding Discussion

  27. A Conceptual Restructuring of the Syllabus 1 • Overview + Basic Concepts of Neurons and Schemas • 1. Modeling the Mirror System: Setting Goals for the Course [Proposal] {Background: TMB Chapter 1} • 2. Charting the Brain 1 [TMB 2.4] • 3. The Brain as a Network of Neurons [TMB Section 2.3] • 9. Schemas for Reaching and Grasping; Affordances [TMB 2.2, 5.3] {Background TMB 2.1, 5.2} • 10. Charting the Brain 2

  28. A Conceptual Restructuring of the Syllabus 2 • Vision • Preprocessing • 4. Visual Preprocessing [TMB 3.3] • 6. Hebbian Learning and Visual plasticity; Self-organizing feature maps; [NSLJ] Kohonen maps • Low-Level Vision • Supplementary reading not covered in lectures: Perceptual and motor schemas: Discussion of visual segmentation based on, e.g., edge and texture cues; Stereoscopic vision [TMB 7.1]; Motion perception and optic flow [TMB 7.2]. • High-Level Vision • 7. Higher level vision: object recognition {Background TMB 5.2} • 18. The MNS1 Model 1: Hand Recognition

  29. A Conceptual Restructuring of the Syllabus 3 • Motor Control • 19. Systems concepts; Feedback and the spinal cord [TMB 3.1, 3.2] • 20. The MNS 1 Model 2: Simulating the kinematics and biomechanics of reach and grasp • Visuo-Motor Integration • 11. The FARS model of control of grasping 1: Population coding • 12. The FARS model of control of grasping 2: Sequence learning and the basal ganglia • 13. The FARS model of control of grasping 3: Working Memory • 21. The MNS1 Model 3: Modeling the Core Mirror Neuron Circuit • 22. Control of eye movements [TMB 6.2] • 23. Basal Ganglia and Control of eye movements - Dominey [NSLbook]

  30. A Conceptual Restructuring of the Syllabus 4 • Adaptive Networks • 5. Adaptive networks: Hebbian learning, Perceptrons; Landmark learning [TMB 3.4] [NSLbook] • 14. Reinforcement learning and motor control; [NSLJ] Conditional motor learning • 15. Adaptive networks: Gradient descent & backpropagation [TMB 8.2] • 24. Dopamine and Sequence Learning • 25. Abstract models of Sequence Learning • Higher Cognitive Functions • 26. Extending the FARS model to mirror neurons and language • Supplementary reading: Memory and Consciousness [TMB 8.3] • Simulation and Neuroinformatics • 8. Introduction to NSL: modules; SCS schematic capture system; Maxselector model [NSLbook] • 16. [NSLJ] Backprop: a. How to run and write the model [NSLbook] • 17. NeuroBench and the NeuroHomology Database

More Related