Internal Models

1. Internal Models Ferdinando A. Mussa-Ivaldi, “Modular features of motor control and learning,” Current opinion in Neurobiology, Vol. 9, 1999, pp. 713-717 The focus on complex dynamical behaviors has been a feature of recent progress in the control of artificial systems. Advances have been recently brought about in neurobiology by dealing with the complexity of ‘real’ motor behaviors. Among these advances is: The formulation of new hypotheses on the building blocks, or “modules” that the central nervous system (CNS) may use to form useful representations for planning and control of actions.

2. The acquisition of complex skills is facilitated by internal models of dynamics. To achieve its typical competence, the motor system must take advantage of experience for going beyond experience itself !!!!!!!!! Internal Model allow us to generate new behaviors and to handle situations that have not yet been encountered.

3. An Example: The task of balancing an inverted pendulum on the hand of a robotic arm: • The balancing task can be accomplished by humans without much effort => copying every movement made by a human expert (Imitation)

4. Imitation approach has been failed: • Even if the hand movements are carefully copied by the robot, the smallest amount of uncertainty associated with sensing and acting resulted in failure. • This does not mean that imitation of successful behavior is not a valid strategy. • The robot learned to act successfully when the data from the observed demonstrations were used not to mimic the expert’s movements but to buildaninternal representation of the dynamics associated with the task. • Imitation is a useful strategy when it provides the learner with a good starting point for the subsequent development of an internal model by a search procedure.

5. Multiple Internal Models • Why a modular structure? • Many situations are derived from a combination of previously experienced contexts. • The use of modular system allows individual modules to participate in motor learning without affecting the motor behaviors already learned by other modules: Adapting a new module (Learning) & De-adaptation • Multiple internal models can be regarded conceptually asmotor primitives or building blocks used to construct intricate motor behaviors with an enormous vocabulary!

6. Mixture of Experts(Jordan & Jacobs, 1991) Gater Stochastic Decision Making pn p1 u1 Expert1 - X Hierarchical structure (HME) p1 X Model 1 Desired Movement (XD) • Gater • Role of Forward model • Switching • Supervised learning un Expertn X - pn X Model n + Desired Motor Command Predicted Motor Command

7. Multiple Paired Forward-Inverse Model

8. Soft Min-Max (Responsibility signal) λn λ1 X u1 - IDM1 FDM1 x1 X λ1 λ1 X Model 1 un - FDMn IDMn xn X Desired Movement (XD) λn λn X X Model n Sensory Feedback (XA) + Feedback Motor Command Efference Copy Feed-forward Motor Command Multiple Paired Forward-Inverse Model M. Kawato, D. Wolpert, “Internal models for motor control,” Sensory Guidance of Movement, John Wiley & Sons, 1998, pp. 291-307

9. Responsibility Estimator (Posterior) λn λ1 u1 x1 - IDM1 FDM1 X λ1 λ1 X X X Likelihood Model Responsibility Predictor Prior Model 1 - Contextual Signal X un xn - FDMn IDMn X λn λn X X Likelihood Model Responsibility Predictor Model n Prior - + Feedback Motor Command UF Efference Copy Sensory Feedback (XA) Actual Movement Feedback Controller - + Movement System UA Desired Movement (XD) Sensory System UF: Feed-forward Motor Command UA: Actual Motor Command Sensory Feedback D. Wolpert, M. Kawato, “Multiple paired forward and inverse models for motor control,” Neural Network, 11, 1998, pp. 1317-1329

11. Neural Networks as Constructive Instruments for Internal Models Past inputs and targets Recurrent networks incorporate system dynamics in their structure They are capable of reach dynamics, and exhibiting memory and context sensitivity Feed-forward networks do not incorporate any dynamics in their structure

12. GRN with embedded memory or order 2 Standard Global Recurrent Network (GRN) LRN with embedded memory or order 2 Standard Locally Recurrent Network (LRN)