Loading in 2 Seconds...

Automation & Robotics Research Institute (ARRI) The University of Texas at Arlington

Loading in 2 Seconds...

- 258 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about 'Automation & Robotics Research Institute (ARRI) The University of Texas at Arlington' - salena

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

### Results comparing the performances of DT-ADHDP and CT-HDP

Moncrief-O’Donnell Endowed Chair

Head, Controls & Sensors Group

Supported by :

NSF - PAUL WERBOS

ARO – JIM OVERHOLT

Automation & Robotics Research Institute (ARRI)The University of Texas at Arlington

ADP for Feedback Control

Talk available online at

http://ARRI.uta.edu/acs

2007 IEEE International Symposium on Approximate Dynamic Programming and Reinforcement Learning

David Fogel, General Chair

Derong Liu, Program Chair

Remi Munos, Program Co-Chair

Jennie Si, Program Co-Chair

Donald C. Wunsch, Program Co-Chair

Automation & Robotics Research Institute (ARRI)

Relevance- Machine Feedback Control

High-Speed Precision Motion Control with unmodeled dynamics, vibration suppression, disturbance rejection, friction compensation, deadzone/backlash control

Industrial

Machines

Military Land

Systems

Vehicle

Suspension

Aerospace

Fuzzy Associative Memory (FAM)

Neural Network (NN)

(Includes Adaptive Control)

Fuzzy Logic Rule Base

NN

NN

Output

Input

Input Membership Fns.

Output Membership Fns.

Input x

Output u

Input x

Output u

Both FAM and NN define a function u= f(x) from inputs to outputs

- FAM and NN can both be used for: 1. Classification and Decision-Making
- 2. Control

NN Includes Adaptive Control (Adaptive control is a 1-layer NN)

- Learning
- Recall
- Function approximation
- Generalization
- Classification
- Association
- Pattern recognition
- Clustering
- Robustness to single node failure
- Repair and reconfiguration

Nervous system cell.

http://www.sirinet.net/~jgjohnso/index.html

VT

WT

s(.)

x1

s(.)

y1

2

s(.)

x2

s(.)

y2

3

s(.)

xn

ym

s(.)

L

outputs

inputs

s(.)

hidden layer

Two-layer feedforward static neural network (NN)

Summation eqs

Matrix eqs

Have the universal approximation property

Overcome Barron’s fundamental accuracy limitation of 1-layer NN

..

Nonlinear Inner Loop

Feedforward Loop

^

f(x)

q

e

r

[I]

Robot System

Kv

v(t)

Robust Control

Term

PD Tracking Loop

Neural Network Robot Controller

Feedback linearization

Universal Approximation Property

qd

Problem- Nonlinear in the NN weights so

that standard proof techniques do not work

Easy to implement with a few more lines of code

Learning feature allows for on-line updates to NN memory as dynamics change

Handles unmodelled dynamics, disturbances, actuator problems such as friction

NN universal basis property means no regression matrix is needed

Nonlinear controller allows faster & more precise motion

Extension of Adaptive Control to nonlinear-in parameters systems

No regression matrix needed

Forward Prop term?

Extra robustifying terms-

Narendra’s e-mod extended to NLIP systems

Backprop terms-

Werbos

Can also use simplified tuning- Hebbian

But tracking error is larger

Force Control

Flexible pointing systems

Vehicle active suspension

SBIR Contracts

Won 1996 SBA Tibbets Award

4 US Patents

NSF Tech Transfer to industry

q

d

NN#1

q

q

q

q

=

=

.

.

r

r

^

.

e

=

F

(x)

r

r

q

q

1

r

r

h

u

i

e

d

K

K

K

K

h

h

h

r

q

q

.

.

d

d

q

q

=

=

q

q

d

d

d

d

^

F

(x)

2

NN#2

Backstepping Loop

Flexible & Vibratory Systems

Add an extra feedback loop

Two NN needed

Use passivity to show stability

Backstepping

..

..

q

q

d

d

Nonlinear FB Linearization Loop

Nonlinear FB Linearization Loop

NN#1

q

q

e

e

q

q

=

=

.

.

r

r

^

^

.

.

e

e

=

=

F

F

(x)

(x)

r

r

q

q

e

e

1

1

r

r

h

u

i

r

r

Robot

Robot

e

d

L

L

1/K

1/K

[

[

I]

I]

K

K

System

System

B1

B1

r

r

i

i

q

q

.

.

d

d

q

q

=

=

q

q

d

d

d

d

Robust Control

Robust Control

^

^

F

F

(x)

(x)

Term

Term

v

v

(t)

(t)

2

2

i

i

NN#2

Backstepping Loop

Tracking Loop

Tracking Loop

Neural network backstepping controller for Flexible-Joint robot arm

Advantages over traditional Backstepping- no regression functions needed

Actor:

Critic:

Deadzone, saturation, backlash

NN in Feedforward Loop- Deadzone Compensation

little critic network

Acts like a 2-layer NN

With enhanced

backprop tuning !

Tune Action NN -

Needed when all states are not measured

NN Observers

i.e. Output feedback

Recurrent NN Observer

Also Use CMAC NN, Fuzzy Logic systems

Fuzzy Logic System

= NN with VECTOR thresholds

Separable Gaussian activation

functions for RBF NN

Tune first layer weights, e.g.

Centroids and spreads-

Activation fns move around

Dynamic Focusing of Awareness

Separable triangular activation

functions for CMAC NN

Elastic Fuzzy Logic- c.f. P. Werbos

Weights importance of factors in the rules

Effect of change of membership function elasticities "c"

Effect of change of membership function spread "a"

Builds its own basis set-

Dynamic Focusing of Awareness

Better Performance

Start with 5x5 uniform grid of MFS

Optimality in Biological Systems

Cell Homeostasis

The individual cell is a complex feedback control system. It pumps ions across the cell membrane to maintain homeostatis, and has only limited energy to do so.

Permeability control of the cell membrane

http://www.accessexcellence.org/RC/VL/GG/index.html

Cellular Metabolism

Optimality in Control Systems Design

Rocket Orbit Injection

Dynamics

Objectives

Get to orbit in minimum time

Use minimum fuel

http://microsat.sm.bmstu.ru/e-library/Launch/Dnepr_GEO.pdf

2. Neural Network Solution of Optimal Design Equations

Nearly Optimal Control

Based on HJ Optimal Design Equations

Known system dynamics

Preliminary Off-line tuning

1. Neural Networks for Feedback Control

Based on FB Control Approach

Unknown system dynamics

On-line tuning

Extended adaptive control

to NLIP systems

No regression matrix

disturbance

z

d

control

Measured

output

y

u

H-Infinity Control Using Neural Networks

Murad Abu Khalaf

System

where

L2 Gain Problem

Find control u(t) so that

For all L2 disturbances

And a prescribed gain g2

Zero-Sum differential Nash game

Successive Solution- Algorithm 1:

Let g be prescribed and fixed.

a stabilizing control with region of asymptotic stability

1. Outer loop- update control

Initial disturbance

2. Inner loop- update disturbance

Solve Value Equation

Inner loop update disturbance

go to 2.

Iterate i until convergence to

with RAS

Outer loop update control action

Go to 1.

Iterate j until convergence to

, with RAS

Cannot solve HJI !!

Consistency equation

For Value Function

CT Policy Iteration for H-Infinity Control

Problem- Cannot solve the Value Equation!

Neural Network Approximation for Computational Technique

Neural Network to approximate V(i)(x)

(Can use 2-layer NN!)

Value function gradient approximation is

Substitute into Value Equation to get

Therefore, one may solve for NN weights at iteration (i,j)

VFA converts partial differential equation into algebraic equation in terms of NN weights

Neural Network Optimal Feedback Controller

Optimal Solution

A NN feedback controller with nearly optimal weights

Cheng Tao

Fixed-Final-Time HJB Optimal Control

Optimal cost

Optimal control

This yields the time-varyingHamilton-Jacobi-Bellman (HJB) equation

Approximating in the HJB equation gives an ODE in the NN weights

Solve by least-squares – simply integrate backwards to find NN weights

Control is

HJB Solution by NN Value Function Approximation

Time-varying weights

Irwin Sandberg

Note that

where is the Jacobian

Policy iteration not needed!

ARRI Research Roadmap in Neural Networks

3. Approximate Dynamic Programming – 2006-

Nearly Optimal Control

Based on recursive equation for the optimal value

Usually Known system dynamics (except Q learning)

The Goal – unknown dynamics

On-line tuning

Optimal Adaptive Control

Extend adaptive control to

yield OPTIMAL controllers.

No canonical form needed.

2. Neural Network Solution of Optimal Design Equations – 2002-2006

Nearly optimal solution of

controls design equations.

No canonical form needed.

Nearly Optimal Control

Based on HJ Optimal Design Equations

Known system dynamics

Preliminary Off-line tuning

1. Neural Networks for Feedback Control – 1995-2002

Extended adaptive control

to NLIP systems

No regression matrix

Based on FB Control Approach

Unknown system dynamics

On-line tuning

NN- FB lin., sing. pert., backstepping, force control, dynamic inversion, etc.

Four ADP Methods proposed by Werbos

Critic NN to approximate:

AD Heuristic dynamic programming

Heuristic dynamic programming

(Watkins Q Learning)

Value

Q function

Dual heuristic programming

AD Dual heuristic programming

Gradient

Gradients

Action NN to approximate the Control

Bertsekas- Neurodynamic Programming

Barto & Bradtke- Q-learning proof (Imposed a settling time)

= the prescribed control input function

Discrete-Time Optimal Control

cost

Value function recursion

Hamiltonian

Optimal cost

Bellman’s Principle

Optimal Control

System dynamics does not appear

Solutions by Comp. Intelligence Community

System

DT HJB equation

Difficult to solve

Few practical solutions by Control Systems Community

Greedy Value Fn. Update- Approximate Dynamic Programming

ADP Method 1 - Heuristic Dynamic Programming (HDP)

Lyapunov eq.

ADP Greedy Cost Update

Simple recursion

For LQR

Underlying RE

Lancaster & Rodman

proved convergence

Paul Werbos

Policy Iteration

For LQR

Underlying RE

Hewer 1971

Initial stabilizing control is needed

Initial stabilizing control is NOT needed

DT HDP vs. Receding Horizon Optimal Control

Forward-in-time HDP

Backward-in-time optimization – RHC

Control Lyapunov Function

- Action Dependent ADP

Define Q function

uk arbitrary

policy h(.) used after time k

Note

Recursion for Q

Simple expression of Bellman’s principle

Q Learning does not need to know f(xk) or g(xk)

For LQR

V is quadratic in x

Q is quadratic in x and u

Control update is found by

so

Control found only from Q function

A and B not needed

Greedy Q Fn. Update - Approximate Dynamic Programming

ADP Method 3. Q Learning

Action-Dependent Heuristic Dynamic Programming (ADHDP)

Greedy Q Update

Model-free ADP

Paul Werbos

Update weights by RLS or backprop.

Model-free policy iteration

Q Policy Iteration

Bradtke, Ydstie,

Barto

Control policy update

Stable initial control needed

Q learning actually solves the Riccati Equation

WITHOUT knowing the plant dynamics

Model-free ADP

Direct OPTIMAL ADAPTIVE CONTROL

Works for Nonlinear Systems

Proofs?

Robustness?

Comparison with adaptive control methods?

ADP for Discrete-Time H-infinity Control

Finding Nash Game Equilbrium

- HDP
- DHP
- AD HDP – Q learning
- AD DHP

ADP for DT H∞ Optimal Control Systems

Disturbance

Penalty output

wk

zk

Control

uk

yk

Measured

output

uk=Lxk

where

Find control uk so that

for all L2 disturbances

and a prescribed gain g2 when the system is at rest, x0=0.

Two known ways for Discrete-time H-infinity iterative solution

Policy iteration for game solution

Requires stable initial policy

ADP Greedy iteration

Does not require a stable initial policy

Both require full knowledge of system dynamics

DT GameHeuristic Dynamic Programming: Forward-in-time Formulation

An Approximate Dynamic Programming Scheme (ADP) where one has the following incremental optimization

which is equivalently written as

Asma Al-Tamimi

Showed that this is equivalent to iteration on the Underlying Game Riccati equation

Which is known to converge- Stoorvogel, Basar

Asma Al-Tamimi

HDP- Linear System Case

Value function update

Solve by batch LS

or RLS

Control update

Control gain

A, B, E needed

Disturbance gain

Q-Learning for DT H-infinity Control:Action Dependent Heuristic Dynamic Programming

Asma Al-Tamimi

- Dynamic Programming: Backward-in-time
- Adaptive Dynamic Programming: Forward-in-time

Linear Quadratic case- V and Q are quadratic

Asma Al-Tamimi

Q learning for H-infinity Control

Q function update

Control Action and Disturbance updates

A, B, E NOT needed

Probing Noise injected to get Persistence of Excitation

Proof- Still converges to exact result

Asma Al-Tamimi

Quadratic Basis set is used to allow on-line solution

and

where

Quadratic Kronecker basis

Q function update

Solve for ‘NN weights’ - the elements of kernel matrix H

Use batch LS or

online RLS

Control and Disturbance Updates

H-inf Q learning Convergence Proofs

Asma Al-Tamimi

- Convergence – H-inf Q learning is equivalent to solving

without knowing the system matrices

- The result is a model free Direct Adaptive Controller that converges to an H-infinity optimal controller
- No requirement what so ever on the model plant matrices

Direct H-infinity Adaptive Control

Compare to Q function for H2 Optimal Control Case

H-infinity Game Q function

Explicit equation for cost – use LS for Critic NN update

Implicit equation for DT control- use gradient descent for action update

Backpropagation- P. Werbos

Standard Neural Network VFA for On-Line Implementation

NN for Value - Critic

NN for control action

(can use 2-layer NN)

HDP

Define target cost function

x1

x2

time

x1

time

Take sample points along a single trajectory

Recursive Least-Squares RLS

Issues with Nonlinear ADP

LS solution for Critic NN update

Selection of NN Training Set

Integral over a region of state-space

Approximate using a set of points

Batch LS

Set of points over a region vs. points along a trajectory

For Linear systems- these are the same

Conjecture- For Nonlinear systems

They are the same under a persistence of excitation condition

- Exploration

Implicit equation for DT control- use gradient descent for action update

Interesting Fact for HDP for Nonlinear systems

Linear Case

must know system A and B matrices

NN for control action

- Note that state internal dynamics f(xk) is NOT needed in nonlinear case since:
- NN Approximation for action is used
- xk+1 is measured

Continuous-Time Optimal Control

System

c.f. DT value recursion,

where f(), g() do not appear

Cost

Hamiltonian

Optimal cost

Bellman

Optimal control

HJB equation

Linear system, quadratic cost -

System:

Utility:

The cost is quadratic

Optimal control (state feed-back):

HJB equation is the algebraic Riccati equation (ARE):

Utility

Cost for any given u(t)

Lyapunov equation

Iterative solution

- Convergence proved by Saridis 1979 if Lyapunov eq. solved exactly
- Beard & Saridis used complicated Galerkin Integrals to solve Lyapunov eq.
- Abu Khalaf & Lewis used NN to approx. V for nonlinear systems and proved convergence

Pick stabilizing initial control

Find cost

Update control

Full system dynamics must be known

LQR Policy iteration = Kleinman algorithm

1. For a given control policy solve for the cost:

2. Improve policy:

- If started with a stabilizing control policy the matrix monotonically converges to the unique positive definite solution of the Riccati equation.
- Every iteration step will return a stabilizing controller.
- The system has to be known.

Lyapunov eq.

Kleinman 1968

Policy Iteration Solution

Policy iteration

This is in fact a Newton’s Method

Then, Policy Iteration is

Frechet Derivative

Policy iteration

Synopsis on Policy Iteration and ADP

Discrete-time

Policy iteration

If xk+1 is measured,

do not need knowledge of f(x) or g(x)

Need to know f(xk) AND g(xk)

for control update

ADP Greedy cost update

Either measure dx/dt

or must know f(x), g(x)

Need to know ONLY

g(x) for control update

What is Greedy ADP for CT Systems ??

Policy Iterations without Lyapunov Equations

- An alternative to using policy iterations with Lyapunov equations is the following form of policy iterations:
- Note that in this case, to solve for the Lyapunov function, you do not need to know the information about f(x).

Measure the cost

Murray, Saeks, and Lendaris

Methods to obtain the solution

- Dynamic programming
- built on Bellman’s optimality principle – alternative form for CT Systems [Lewis & Syrmos 1995]

Solving for the cost – Our approach

For a given control

The cost satisfies

c.f. DT case

f(x) and g(x) do not appear

LQR case

Optimal gain is

What to do about the tail – issues in Receding Horizon Control

Policy Evaluation – Critic updateLet K be any state feedback gain for the system (1). One can measure the associated cost over the infinite time horizon

where is an initial infinite horizon cost to go.

Now Greedy ADP can be defined for CT Systems

Solving for the cost – Our approachCT ADP Greedy iteration

Control policy

Cost update

LQR

A and B do not appear

Control gain update

B needed for control update

Implement using quadratic basis set

- No initial stabilizing control needed

u(t+T) in terms of x(t+T) - OK

Direct Optimal Adaptive Control for Partially Unknown CT Systems

Algorithm Implementation

Measure cost increment by adding V as a state. Then

The Critic update

can be setup as

Evaluating for n(n+1)/2 trajectory points, one can setup a least squares problem to solve

Or use recursive Least-Squares along the trajectory

a strange pseudo-discretized RE

c.f. DT RE

Analysis of the algorithmwith

For a given control policy

Greedy update is equivalent to

This extra term means the initial

Control action need not be stabilizing

When ADP converges, the resulting P satisfies the Continuous-Time ARE !!

Analysis of the algorithm

Lemma 2. CT HDP is equivalent to

ADP solves the CT ARE without knowledge of the system dynamics f(x)

WITHOUT knowing the plant dynamics

Model-free ADP

Direct OPTIMAL ADAPTIVE CONTROL

Works for Nonlinear Systems

Proofs?

Robustness?

Comparison with adaptive control methods?

k

0

1

2

3

4

5

Gain update (Policy)

Control

t

Sample periods need not be the same

Continuous-time control with discrete gain updates

Higher Central Control of Afferent Input

Descending tracts from the brain influence not only motor neurons but also the gamma-neurons which regulate sensitivity of the muscle spindle.

Central control of end-organ sensitivity has been demonstrated.

Many brain structures exert control of the first synapse in ascending systems.

Role of cerebello rubrospinal tract and Purkinje Cells?

T.C. Rugh and H.D. Patton, Physiology and Biophysics, p. 213, 497,Saunders, London, 1966.

Small Time-Step Approximate Tuning for Continuous-Time Adaptive Critics

Baird’s Advantage function

Advantage learning is a sort of first-order approximation to our method

Submitted to IJCNN’07 Conference

Asma Al-Tamimi and Draguna Vrabie

System, cost function, optimal solution

System – power plant Cost

CARE:

Wang, Y., R. Zhou, C. Wen - 1993

CT HDP results

The state measurements were taken at each 0.1s time period.

A cost function update was performed at each 1.5s.

For the 60s duration of the simulation a number of 40 iterations (control policy updates) were performed.

Convergence of the P matrix parameters for CT HDP

The discrete version was obtained by discretizing the continuous time model using zero-order hold method with the sample time T=0.01s.

Continuous-time used only 40 iterations!

DT ADHDP resultsThe state measurements were taken at each 0.01s time period.

A cost function update was performed at each .15s.

For the 60s duration of the simulation a number of 400 iterations (control policy updates) were performed.

Convergence of the P matrix parameters for DT ADHDP

- CT HDP
- Partially model free (the system A matrix is not required to be known)
- DT ADHDP – Q learning
- Completely model free

The DT ADHP algorithm is computationally more intensive than the CT HDP since it is using a smaller sampling period

IEEE Transactions on Systems, Man, & Cybernetics- Part B

Special Issue on

Adaptive Dynamic Programming and Reinforcement Learning

in Feedback Control

George Lendaris

Derong Liu

F.L Lewis

Papers due 1 August 2007

Download Presentation

Connecting to Server..