Movement imitation linking perception and action
Download
1 / 53

Movement Imitation: Linking Perception and Action - PowerPoint PPT Presentation


  • 115 Views
  • Uploaded on

Movement Imitation: Linking Perception and Action. Advanced Topics in Computer Vision, 2004 Lior Noy Department of Computer Science and Applied Mathematics Weizmann Institute of Science. Movement Imitation - Example. Imitation: Linking Perception and Action.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Movement Imitation: Linking Perception and Action' - ailish


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Movement imitation linking perception and action
Movement Imitation: Linking Perception and Action

Advanced Topics in Computer Vision, 2004

Lior Noy

Department of Computer Science and Applied Mathematics

Weizmann Institute of Science



Imitation: Linking Perception and Action

semantic world(objects, actions)

Perception

Imitation

Action

realm of raw-data (pixels, muscles activation)


Outline
Outline

1. MovementImitation

2. Programming By Demonstration

  • 3. Robotic Movement Imitation

  • Primitives Based Approach (Mataric’)

  • Real Time Tracking (“mirror-game”) (Ude et al.)

4. Direct Perception and Imitation


A variety of probes into imitation
A Variety of Probes into Imitation

Developmental psychology

Ethology

Imitation

Human Brain Imaging

Cognitive psychology

Neurophysiology

Robotics


Evaluating imitation robot following in a hilly environment
Evaluating ImitationRobot Following in a Hilly Environment




Programming By Demonstration (PbD)

Methods to program a robot

  • Human Programming

  • Reinforcement Learning

  • Programming by Demonstration


Programming by demonstration pbd applications
Programming By Demonstration (PbD)Applications

  • Navigation

  • Locomotion

  • Playing air-hockey

  • Manipulating blocks

  • Balancing a pole

  • Hitting a tennis-serve

  • Grasping unfamiliar objects

  • Imitating dancing movement


Pbd application example
PbD – Application Example

The “Golden Maze”


Pbd application example1
PbD – Application Example

Playing Air-Hockey


PbD – Application Example

Box Manipulations


Three Approaches for PbD

  • Symbolic

  • Control-Based

  • Statistical


Symbolic Approach for PbD

  • Analyze observed actions in terms of sub-goals

  • Match actions needed to fulfill these sub-goals

  • Createa symbolic description of the environment ( ”object A is above object B” )

  • Learn a series of symbolic if-thenrules ( ”if object A is aboveobject B then grasp-object[ object B ]” )


Example symbolic approach for pbd
Example: Symbolic Approach for PbD

(Kunyushi et al., 1994)

… but how do you symbolically describes “hitting a tennis serve”?


Control-Based Approach for PbD

  • No symbolic parsing of perceived actions

  • Assume a pre-defined control policy

  • Acquire needed parameters from observation


Forward Models

motor commands

jointangles

end-effectorposition

Inverse Models

Control-Based Approach Inverse Models

Sometime assume known inverse models (converting desired effect to needed commands)


Example control based approach for pbd
Example: Control-Based Approach for PbD

(Schaal, 2003)

Tennis

movie


Statistical approach for pbd
Statistical Approach for PbD

  • No prior assumption on used control policy

  • Statistically match perception and action

  • Can this be done? More on this later…


Example statistical approach for pbd

PCA

Example: Statistical Approach for PbD

(Asada, 1995)


Example: Statistical Approach for PbD

  • Learning:

  • Perform random action A(i)

  • Record resulted optical flow f(i)

  • Compute principal-component p1(i), p2(i)

  • Learn the connection A(i) – {p1(i), p2(i)}


Outline1
Outline

1. MovementImitation

2. Programming By Demonstration

  • 3. Robotic Movement Imitation

  • Primitives Based Approach (Mataric’)

  • Real Time Tracking (“mirror-game”) (Ude et al.)

4. Direct Perception and Imitation


Pbd for movement imitation pre cursor 1 cartoons retargeting
PbD for Movement ImitationPre-Cursor 1: Cartoons Retargeting

(Bregler et al., 2002)


y

x

Cartoons RetargetingTwo Types of Deformations

  • Affine deformation

  • Key shape deformation


Cartoons Retargeting - Results

More on: http://www.cs.weizmann.ac.il/~hassner/cv03/“Animating human motion”, Speakers : Simon Adar, Yoram Atir


Pbd for movement imitation pre cursor 2 guided movement synthesis
PbD for Movement ImitationPre-Cursor 2: Guided Movement Synthesis

(Zelnik-Manor, Hassner & Irani, 2004)


Event based analysis of video
Event-Based Analysis of Video

(Zelnik-Manor & Irani, 2001)


Guided movement synthesis a k a movement imitation
Guided Movement Synthesis(a.k.a. “Movement Imitation”?)


PbD for Movement ImitationPre-Cursor 2: Movement Synthesis


Pbd for movement imitation case study primitive based approach

X1X2...Xn

J1J2...Jm

movementprimitive1

movementprimitive2

=

=

...

movementprimitive K

PbD for Movement ImitationCase Study: Primitive-Based Approach

The Problem: How to convert visual input to motor output?

A Possible Solution: Use a common, sparse representation: sensory-motor primitives.

… but what primitives to use?


Movement imitation using sensory motor primitives
Movement Imitation Using Sensory-Motor Primitives

Motor primitives:

Sequences of action that accomplish a complete goal-directed behavior.

Examples:1. Move hand in “straight line”, “parabola” (Felix…).2. Perform “grasping”, “a tennis serve”.


Imitation Learning Using Sensory-Motor Primitives

(Schaal, Ijspeert & Billard, 2003)


Inspiration for Using Sensory-Motor Primitives

  • Evidence for:

  • Coding of goal-directed actions.

  • Shared representations of perception and action.

  • Example – Mirror Neurons.

(Rizzolatti et al., 2002; Gallese et al. 1996)


Movement imitation using sensory motor primitives1
Movement Imitation Using Sensory-Motor Primitives

  • General Principles:

  • Selective attention focusing on end-points movements.

  • Sensory-motor primitives as integrative representation.

  • Learning new skills as compositions of primitives.

  • Experimental test-beds.

(Mataric’,1998)


What sensory motor primitives to use
What Sensory-Motor Primitives to Use?

InnatePre-defined control policies (e.g., central pattern generators)

LearnedUn-supervised clustering (using PCA, Isomap )

Primitives

End-Points Space(“visual space”)

Joints Space(“motor space”)


Experiment in imitation using perceptuo motor primitives weber jenkins mataric 2001
“Experiment in Imitation Using Perceptuo-Motor Primitives”, (Weber, Jenkins & Mataric’,2001)

  • Extract hand (end-point) movements.

  • Perform Vector-Quantization to get invariant representation.


“Experiment in Imitation Using Perceptuo-Motor Primitives”

  • Classify movement to primitives (line, arc, circle).

  • Group adjacent similar primitives.


“Experiment in Imitation Using Perceptuo-Motor Primitives”

  • Determine primitives parameters.

  • Project to ego-centric space.


“Experiment in Imitation Using Perceptuo-Motor Primitives”

(Weber, Jenkins & Mataric’,2001)


Pbd for movement imitation case study real time tracker
PbD for Movement Imitation Primitives”Case Study: Real-Time Tracker

  • The Goal:Mimic movements in real-time

  • The Problem:

    • Large amount of data to process (6 MB/Sec)

    • Need “continuous success”

  • The Solution:

    • Probabilistic approach to prevent excessive data interactions

(Ude et al.,2001)


Real time visual system for interaction with humanoid robot ude shibata atkeson 2001
“Real-Time Visual System for Interaction with Humanoid Robot”(Ude, Shibata & Atkeson, 2001)

Estimate positions of tracked “blobs” in the image

Compute 3D coordinates of tracked objects using stereo

Transform into via-points for robot hand trajectory

Compute motor commands from desired trajectory


Real time tracker tracking blobs in a bayesian setting
Real-Time Tracker Robot”Tracking “Blobs” In a Bayesian Setting

probability for the pixel at location u to have Intensity Iu

Given the process k

a-priori probability for process k


Real time tracking minimize log likelihood
Real-Time Tracking Robot”Minimize Log-Likelihood

overall probabilityto observe image I

Goal: determine the parameters that are most likely to produce this image – Maximal Likelihood Problem.

computationally easier to minimize the negative log likelihood


Real time tracking minimize log likelihood1
Real-Time Tracking Robot”Minimize Log-Likelihood

Find minimum (using Lagrange Multipliers) and get:

probability that pixel ustems from process l


Real-Time Tracking Robot”Find Probabilities Parameters

The above equations are solved iteratively by the Expectation-Minimization (EM) algorithm

  • Expectation stage:

  • compute Pu,l using the current estimate for Ө and ω.

  • Minimization stage:

  • compute new Ө and ω assuming Pu,l are constant.

from probabilities of pixels to belong to a certain process (e.g. – the human hand) …


Real time visual system for interaction with humanoid robot
“Real-Time Visual System for Robot”Interaction with Humanoid Robot”

… to object locations


Real time tracking general stages
Real-Time Tracking Robot”General Stages

Estimate positions of tracked “blobs” in the image

Compute 3D coordinates of tracked objects using stereo

Transform into via-points for robot hand trajectory

Compute motor commands from desired trajectory


Real time tracking estimate trajectories with b splines
Real-Time Tracking Robot”Estimate Trajectories with B-splines


Real time tracking results
Real-Time Tracking - Results Robot”

Robot Compliance Movie


References
References Robot”

“Vision-Based Robot Learning for Behavior Acquisition”M. Asada, T. Nakamura, and K. Hosoda.Proc. of IEEE International Conference on Intelligent Robots And Systems 1995 (IROS '95) Workshop on Vision for Robots, pp.110-115, 1995.

“Turning to the masters: Motion capturing cartoons”Bregler C, Loeb L, Chuang E, Deshpande HACM TRANSACTIONS ON GRAPHICS 21 (3): 399-407 JUL 2002

“Movement, activity and action: The role of knowledge in the perception of motion”Bobick AFPHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY OF LONDON SERIES B-BIOLOGICAL SCIENCES 352 (1358): 1257-1265 AUG 29 1997

“Challenges in Building Robots That Imitate People”,Breazeal C. and Scassellati B, in "Imitation in Animals and Artifacts", Kerstin Dautenhahn and Chrystopher Nehaniv, eds.The MIT Press, 2002.

“Action recognition in the premotor cortex”Gallese V, Fadiga L, Fogassi L, Rizzolatti GBRAIN , 119: 593-609 Part 2 APR 1996

“Learning by watching - extracting reusable task knowledge from visual observation of human-performance”Kuniyoshi Y, Inaba M, Inoue HIEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION,10 (6): 799-822 DEC 1994

“Sensory-Motor Primitives as a Basis for Learning by Imitation: Linking Perception to Action and Biology to Robotics.” Maja J Mataric, in "Imitation in Animals and Artifacts", Kerstin Dautenhahn and Chrystopher Nehaniv, eds., MIT Press, 2002, 392-422


References1
References Robot”

“From mirror neurons to imitation: facts and speculations”, Rizzolatti G, Fadiga L, Fogassi L and Gallese V, in: Meltzoff AN and Prinz W (Eds.) "The imitative mind: development, evolution, and brain bases", New York: Cambridge University Press, 2002

“Computational approaches to motor learning by imitation”Schaal S, Ijspeert A, Billard APHILOSOPHICAL TRANSACTIONS OF THE ROYAL SOCIETY OF LONDON SERIES B-BIOLOGICAL SCIENCES, 358 (1431): 537-547 MAR 29 2003

“Movement planning and imitation by shaping nonlinear attractors” Schaal S, PROCEEDINGS OF THE 12TH YALE WORKSHOP ON ADAPTIVE AND LEARNING SYSTEMS 2003

“Robots that imitate humans” Scassellati B. Breazeal C. Trends in Cognitive Science, 6(11):481 487, November 2002.

“Real-time visual system for interaction with a humanoid robot”, Ude A., Shibata T. and Atkeson C. G., Robotics and Autonomous Systems, 37:115 125, 2001.

Stefan Weber, Odest C. Jenkins, and Maja J. Mataric´. "Imitation Using Perceptual and Motor Primitives". In International Conference on Autonomous Agents, pages 136-137, Barcelona, Spain, Jun 2000

"Event-Based Analysis of Video “,Zelnik-Manor L. and  Irani M.,IEEE CONFERENCE ON COMPUTER VISION AND  PATTERN RECOGNITION, December 2001 (CVPR'01).


Perception? Action? Robot”

“The great end of life is not knowledge but action.”

(Thomas H. Huxley)


ad