Bio-inspired Robot Motor Learning

1. Bio-inspired Robot Motor Learning F.C. Park Seoul National University

2. The State-of-the-Art Sony Q-Rio Demonstration

3. Some Questions • Is there a “physical intelligence” behind these movements, e.g., are they based on certain “primitives” that can be reused and generalized to new contexts (learning)? Or are they just manually created movements? • How much “feedforward” versus “feedback” is used in the motor program? What, if any, distinctions can be made between “planning” and “control”?

4. Lessons from Human Motor Control • Bernstein’s degrees of freedom problem and motor synergies • Movement learning as a transition process from closed-loop to open-loop control. • Alain Berthoz’s theories about perception as simulated action—the brain as a “simulator” that anticipates (Alain Berthoz,The Brain’s Sense of Movement)

5. Topics of this (Very Brief) Talk • Describe my attempts to build a general framework for movement generation • Creating feedforward plans: dynamically optimal movements. • Movement primitives using principal components and HMMs • Primitives for balancing and posture

6. State-of-the-Art in Robotics • Operational space control and elastic roadmaps (Khatib, Brock): Potential functions in operational space are used to define goal “attractors”, avoid collisions, etc., all the while exploiting null space dynamics to, e.g., maintain posture. • Rapidly-exploring random trees (RRT) (Kuffner, LaValle): Incrementally growing a randomized search tree by applying control inputs over short intervals.

7. Elastic Roadmap Implementation

8. Is There Room for Improvement? • Potential functions only capture feedback; little if any feedforward or memory-based aspects of movement. • Potential functions for, e.g.,dynamic balance, are not easily formulated (ZMP depends on COM acceleration coordinates) • RRT methods are sensitive to the choice of metric, and nearest neighbor computations are still a burden • The resulting motions are not necessarily optimal or reusable.

9. Creating Optimal Movements • Minimize an integral cost functional of the form subject to the dynamic equations and various boundary conditions and constraints on the state and control

10. Example: Minimum Torque Motions • Numerical algorithms like steepest descent, Newton-type methods require gradient (and sometimes Hessian) information: • Dynamic equations must be differentiated. • Without analytic gradients (and sometimes Hessians), convergence is unreliable.

11. Robot Dynamics Equations • Dynamics for a standard 6-axis industrial robot:

12. Robot Dynamics Equations (p2)

13. Robot Dynamics Equations (p3)

14. Robot Weightlifting Demo (w/J. Bobrow) PUMA Robot Weightlifting Demo

15. Why Can’t Robots Jump like Humans? • The human biarticular muscle is important in the generation of fast explosive human movements. • Spring-like behavior (energy restoring and release) is achieved by the elastic property of muscle, and efficient energy transfer by extension over two joints improves, e.g., jumping performance. • Biarticular structures for robots are intriguing. < monoarticular muscle > < biarticular muscle >

16. A Biarticular Robot Leg (J. Babic) (1) Initially acts as passive prismatic joint (2) When biarticular mechanism is activated, robot becomes redundantly actuated by biarticular force (3) Immediately after robot pushes off the ground, the biarticular actuator is deactivated < Robot model>

17. Optimal Jumps for Biarticular Legs • Dynamics model involves not only a closed chain with redundant actuation and spring element, but also contact models between the feet and ground. • Lemke’s algorithm for the LCP formulation to solve contact problem with friction • Time stepping schemes to reduce integration error • Optimization involves diverse parameters in addition to the actuator inputs: • Spring stiffness, • Biarticular actuator activation angle

18. Maximum Height Jumps • Maximum height vertical jump • Nearly 23% higher (comparing the conventional robot) • Dynamics constraints considered. • Joint torque limits, joint velocity limit Conventional robot Biarticular legged robot (k=10000N/m, ) < Initial motion > < Opimized motion >

19. Maximum Distance Broad Jumps • Maximum distance broad jump • Nearly 20% longer (comparing the conventional robot) • Dynamics constraints considered. • Joint torque limits, joint velocity limit Conventional robot Biarticular legged robot (k=5000N/m, ) < Optimized motion > < Initial motion >

20. Optimization: Some Shortcomings • Computationally expensive, not real-time. • Results are not readily re-usable. • Not clear how to, e.g., perform complex movement sequences, or optimally perform multiple tasks.

21. Reusability: Movement Primitives • Movement primitives for reaching motions are extracted by a principal component analysis of training data (e.g., dynamically optimal movements, motion capture data) • The primitives can serve as basis functions for linear interpolation. • The primitives can also serve as basis functions for dynamic optimization.

22. PCA of Arm-Reaching Motions • Repeated trials of an arm motion (e.g., lifting) are performed PCA Joint angle trajectories of sample motions First 4 principal components for each joint

23. Sample Primitive Database

24. Example Reaching Movements • Optimal movements actually appear less natural than the PCA-based (suboptimal) movements Optimal Movements Sub-Optimal Movements

25. Primitives for Balancing & Posture • Given: • A legged robot standing on some subset of its legs • An input reference joint trajectory, possibly dynamically unstable • Objective:Adjust the reference joint trajectory in real-time to maintain posture while • tracking the input motion • withstanding external disturbances

26. An Optimization Formulation • General balancing can be cast as a second-order cone programming (SOCP) problem:

27. LCLP LCQP QCQP SOCP NLCP ⊂ ⊂ ⊂ A Hierarchy of Optimization Problems • A partial hierarchy of convex optimization problems: • SOCP is a slightly generalized version of QCQP • General formulation of SOCP problem (c = 0 leads to QCQP): • General features of convex programming problems: • Any local optimum is also a global optimum. • There exist many algorithms and software, e.g., MOSEK, free software by Boyd and Vandenberghe, etc. ⊂

28. Maintaining Posture in Real-Time • The original motion is a standing posture. • The results closely resemble a human’s reaction. The units of are and

29. Kicking while Maintaining Balance Original (unstable) kick Stabilized kick ZMP trajectories COG trajectories

30. Towards a Unified Motor Control Theory • Further inspiration from human motor control: • Minimum Variance Principle (Wolpert), Minimum Intervention Principle (Todorov and Jordan) • The Minimum Attention Paradigm (Brockett): • Often imprecise control is as good as precise control—reducing control implementation costs is more important. • Defining control implementation costs: the easiest control is a constant control. • One possible optimal control formulation: