Learning early constructions (Chang, Mok). The ICSI/Berkeley Neural Theory of Language Project. ECG. Moving from Spatial Relations to Verbs. Open class vs. closed class How do we represent verbs (say of hand motion) Can we build models of verbs based on motor control primitives?
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Multiple, chronically implanted, intracranial microelectrode arrays would be used to sample the
activity of large populations of single cortical neurons simultaneously. The combined activity of
these neural ensembles would then be transformed by a mathematical algorithm into continuous
three-dimensional arm-trajectory signals that would be used to control the movements of a
robotic prosthetic arm. A closed control loop would be established by providing the subject with
both visual and tactile feedback signals generated by movement of the robotic arm.
Rizzolatti et al. 1998
Rizzolatti et al. 1998
Gallese and Goldman, TICS 1998
Observed: A is Precision Grip
B is Whole Hand Prehension
Action: C: precision grip
D: Whole Hand Prehension
(Gallese et al. Brain 1996)
Umiltà et al. Neuron 2001
Kohler et al. Science (2002)
PMv (F5ab) – AIP Circuit
“grasp” neurons – fire in relation to movements of hand prehension necessary to grasp object
F4 (PMC) (behind arcuate) – VIP Circuit
transforming peri-personal space coordinates so can move toward objects
PMv (F5c) – PF Circuit F5c
different mirror circuits for grasping, placing or manipulating object
Together suggest cognitive representation of the grasp, active in action imitation and action recognition
General Purpose, and Action-Location
Mirror: Fadiga et al. 1995; Grafton et al. 1996;
Rizzolatti et al. 1996; Cochin et al. 1998;
Decety et al. 1997; Decety and Grèzes 1999;
Hari et al. 1999; Iacoboni et al. 1999;
Buccino et al. 2001.
General Purpose: Perani et al. 1995; Martin et al.
1996; Grafton et al. 1996; Chao and Martin 2000.
Action-Location: Bremmer, et al., 2001.
Ways to grab this “thing”
“It’s a mug”
PFCFARS (Fagg-Arbib-Rizzolatti-Sakata) Model
AIP extracts the set of affordances for an attended object.These affordances highlight the features of the object relevant to physical interaction with it.
Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
The premotor and parietal areas, rather than having
separate and independent functions, are neurally integrated
not only to control action, but also to serve the function of
constructing an integrated representation of:
Actions, together with
objects acted on, and
locations toward which actions are directed.
In these circuits sensory inputs are transformed in order to
accomplish not only motor but also cognitive tasks, such as
space perception and action understanding.
Result of Firing
when two of the neurons fire, the third also fires
We understand utterances by mentally simulating their content.
meaning : Walk-Action
selfm.time before Context.speech-time
“Harry walked into the cafe.”
CAFESimulation-based language understanding
food toys misc. people sound emotion action prep. demon. social
Words learned by most 2-year olds in a play school (Bloom 1993)
A model of children learning their first verbs.
Assumes parent labels child’s actions.
Child knows parameters of action, associates with word
Program learns well enough to:
1) Label novel actions correctly
2) Obey commands using new words (simulation)
System works across languages
Mechanisms are neurally plausible.
How do I specify all the effects