1 / 45

- PowerPoint PPT Presentation

Introduction To Non–linear Optimization. PART I. Optimization Tree. Figure 1: Optimization tree. What is Optimization?. Optimization is an iterative process by which a desired solution (max/min) of the problem can be found while satisfying all its constraint or bounded conditions.

Related searches for

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

PowerPoint Slideshow about '' - syshe

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Figure 1: Optimization tree.

• Optimization is an iterative process by which a desired solution

• (max/min) of the problem can be found while satisfying all its

• constraint or bounded conditions.

Figure 2: Optimum solution is found while satisfying its constraint (derivative must be zero at optimum).

• Optimization problem could be linear or non-linear.

• Non –linear optimization is accomplished by numerical ‘Search

Methods’.

• Search methods are used iteratively before a solution is achieved.

• The search procedure is termed as algorithm.

• Linear problem – solved by Simplex or Graphical methods.

• The solution of the linear problem lies on boundaries of the feasible

region.

Figure 3: Solution of linear problem

Figure 4: Three dimensional solution of non-linear problem

• Non-linear problem solution lies within and on the boundaries of the

feasible region.

Subject to:X1 + X2 ≤ 1500.25 X1 + 0.5 X2 ≤ 50X1 ≥ 50X2 ≥ 25X1 ≥0, X2 ≥0

Fundamentals of Non-Linear Optimization

• Single Objective function f(x)

• Maximization

• Minimization

• Design Variables, xi , i=0,1,2,3…..

• Constraints

• Inequality

• Equality

Figure 5: Example of design variables and constraints used in non-linear optimization.

• Optimal points

• Local minima/maxima points: A point or Solution x* is at local point

if there is no other x in its Neighborhood less than x*

• Global minima/maxima points: A point or Solution x** is at global

point if there is no other x in entire search space less than x**

Figure 7: Local point is equal to global point if

the function is convex.

Figure 6: Global versus local optimization.

• Function f is convex if f(Xa) is less than value of the corresponding

• point joining f(X1) and f(X2).

• Convexity condition – Hessian 2nd order derivative) matrix of

• function f must be positive semi definite ( eigen values +ve or zero).

Figure 8: Convex and nonconvex set

Figure 9: Convex function

• Slop or gradient of the objective function f – represent the

• direction in which the function will decrease/increase most rapidly

• Taylor series expansion

• Jacobian – matrix of gradient of f with respect to several variables

• First order Condition (FOC)

• Hessian – Second derivative of f of several variables

• Second order condition (SOC)

• Eigen values of H(X*) are all positive

• Determinants of all lower order of H(X*) are +ve

• Deterministic - specific rules to move from one iteration to next ,

• Stochastic – probalistic rules are used for subsequent iteration

• Optimal Design – Engineering Design based on optimization algorithm

• Lagrangian method – sum of objective function and linear

combination of the constraints.

• Deterministic

• Direct Search – Use Objective function values to locate minimum

• Gradient Based – first or second order of objective function.

• Minimization objective function f(x) is used with –ve sign –

• f(x) for maximization problem.

• Single Variable

• Newton – Raphson is Gradient based technique (FOC)

• Golden Search – step size reducing iterative method

• Multivariable Techniques ( Make use of Single variable Techniques

• specially Golden Section)

• Unconstrained Optimization

• a.) Powell Method – Quadratic (degree 2) objective function polynomial is

• b.) Gradient Based – Steepest Descent (FOC) or Least Square minimum

• (LMS)

• c.) Hessian Based -Conjugate Gradient (FOC) and BFGS (SOC)

• Constrained Optimization

• a.) Indirect approach – by transforming into unconstrained

• problem.

• b.) Exterior Penalty Function (EPF) and Augmented Lagrange

• Multiplier

• c.) Direct Method Sequential Linear Programming (SLP), SQP and

• Steepest Generalized Reduced Gradient Method (GRG)

• Figure 10: Descent Gradient or LMS

• Global Optimization – Stochastic techniques

• Simulated Annealing (SA) method – minimum energy principle of cooling metal crystalline structure

• Genetic Algorithm (GA) – Survival of the fittest

• principle based upon evolutionary theory

J is the cost function to be minimized in two

dimension

The contours of the J paraboloid shrinks as it is

decrease

function retval = Example6_1(x)

% example 6.1

retval = 3 + (x(1) - 1.5*x(2))^2 + (x(2) - 2)^2;

>> SteepestDescent('Example6_1', [0.5 0.5], 20, 0.0001, 0, 1, 20)

Where

[0.5 0.5] -initial guess value

20 -No. of iteration

0.001 -Golden search tol.

0 -initial step size

1 -step interval

20 -scanning step

>> ans

2.7585 1.8960

Figure 11: Multivariable Gradient based optimization

Figure 12: Steepest Descent

MATLAB Optimization Toolbox

PART II

• Introduction

• Function Optimization

• Optimization Toolbox

• Routines / Algorithms available

• Minimization Problems

• Unconstrained

• Constrained

• Example

• The Algorithm Description

• Multiobjective Optimization

• Optimal PID Control Example

• Optimization concerns the minimization or maximization of

functions

• Standard Optimization Problem:

Subject to:

Equality Constraints

Inequality Constraints

Side Constraints

Where:

is the objective function, which measure and evaluate the performance of a system. In a standard problem, we are minimizing the function. For maximization, it is equivalent to minimization of the –ve of the objective function.

is a column vector of design variables, which can

affect the performance of the system.

• Constraints – Limitation to the design space. Can be linear or

nonlinear, explicit or implicit functions

Equality Constraints

Inequality Constraints

Most algorithm require less than!!!

Side Constraints

• Is a collection of functions that extend the capability of MATLAB.

• The toolbox includes routines for:

• Unconstrained optimization

• Constrained nonlinear optimization, including goal attainment

• problems, minimax problems, and semi-infinite minimization

• problems

• Nonlinear least squares and curve fitting

• Nonlinear systems of equations solving

• Constrained linear least squares

• Specialized algorithms for large scale problems

• Most of these optimization routines require the definition of an M-

file containing the function, f, to be minimized.

• Maximization is achieved by supplying the routines with –f.

• Optimization options passed to the routines change optimization

parameters.

• Default optimization parameters can be changed through an

options structure.

• Consider the problem of finding a set of values [x1 x2]T that

solves

• Steps:

• Create an M-file that returns the function value (Objective

• Function). Call it objfun.m

• Then, invoke the unconstrained minimization routine. Use fminunc

function f = objfun(x)

f=exp(x(1))*(4*x(1)^2+2*x(2)^2+4*x(1)*x(2)+2*x(2)+1);

Objective function

Starting with a guess

x0 = [-1,1];

options = optimset(‘LargeScale’,’off’);

[xmin,feval,exitflag,output]=

fminunc(‘objfun’,x0,options);

Optimization parameters settings

Output arguments

Input arguments

xmin =

0.5000 -1.0000

feval =

1.3028e-010

exitflag =

1

output =

iterations: 7

funcCount: 40

stepsize: 1

firstorderopt: 8.1998e-004

algorithm: 'medium-scale: Quasi-Newton line search'

Minimum point of design variables

Objective function value

Exitflag tells if the algorithm is converged.

If exitflag > 0, then local minimum is found

Some other information

More on fminunc– Input

fminunc(fun,x0,options,P1,P2,…)

fun : Return a function of objective function.

x0 : Starts with an initial guess. The guess must be a vector

of size of number of design variables.

Option : To set some of the optimization parameters. (More after few slides)

P1,P2,… : To pass additional parameters.

More on fminunc– Output

fminunc(fun,x0,options,P1,P2,…)

• xmin : Vector of the minimum point (optimal point). The size is the number of design variables.

• feval : The objective function value of at the optimal point.

• exitflag : A value shows whether the optimization routine is terminated successfully. (converged if >0)

• Output : This structure gives more details about the optimization

• hessian : The hessian value of at the optimal point

Options Setting – optimset

Options =

optimset(‘param1’,value1, ‘param2’,value2,…)

• The routines in Optimization Toolbox has a set of default

optimization parameters.

• However, the toolbox allows you to alter some of those

parameters, for example: the tolerance, the step size, the gradient

or hessian values, the max. number of iterations etc.

• There are also a list of features available, for example: displaying

the values at each iterations, compare the user supply gradient or

hessian, etc.

• You can also choose the algorithm you wish to use.

Options =

optimset(‘param1’,value1, ‘param2’,value2,…)

• Type help optimset in command window, a list of options

setting available will be displayed.

• How to read? For example:

LargeScale - Use large-scale algorithm if possible [ {on} | off ]

The default is with { }

Value (value1)

Parameter (param1)

Options =

optimset(‘param1’,value1, ‘param2’,value2,…)

LargeScale - Use large-scale algorithm if possible [ {on} | off ]

Since the default is on, if we would like to turn off, we just type:

Options = optimset(‘LargeScale’, ‘off’)

and pass to the input of fminunc.

Highly recommended to use!!!

• Display - Level of display [ off | iter | notify | final ]

• MaxIter - Maximum number of iterations allowed [ positive integer ]

• TolCon - Termination tolerance on the constraint violation [

positive scalar ]

• TolFun - Termination tolerance on the function value [ positive

scalar ]

• TolX - Termination tolerance on X [ positive scalar ]

fminuncandfminsearch

• fminunc uses algorithm with gradient and hessian information.

• Two modes:

• Large-Scale: interior-reflective Newton

• Medium-Scale: quasi-Newton (BFGS)

• Not preferred in solving highly discontinuous functions.

• This function may only give local solutions..

• fminsearch is generally less efficient than fminunc for

problems of order greater than two. However, when the problem

is highly discontinuous, fminsearch may be more robust.

• This is a direct search method that does not use numerical or

• This function may only give local solutions.

Vector of Lagrange

Multiplier at optimal point

function f = myfun(x)

f=-x(1)*x(2)*x(3);

Subject to:

For

Create a function call nonlcon which returns 2 constraint vectors [C,Ceq]

function [C,Ceq]=nonlcon(x)

C=2*x(1)^2+x(2);

Ceq=[];

Remember to return a null

Matrix if the constraint does

not apply

Initial guess (3 design variables)

x0=[10;10;10];

A=[-1 -2 -2;1 2 2];

B=[0 72]';

LB = [0 0 0]';

UB = [30 30 30]';

[x,feval]=fmincon(@myfun,x0,A,B,[],[],LB,UB,@nonlcon)

CAREFUL!!!

fmincon(fun,x0,A,B,Aeq,Beq,LB,UB,NONLCON,options,P1,P2,…)

Warning: Large-scale (trust region) method does not currently solve this type of problem, switching to medium-scale (line search).

> In D:\Programs\MATLAB6p1\toolbox\optim\fmincon.m at line 213

In D:\usr\CHINTANG\OptToolbox\min_con.m at line 6

Optimization terminated successfully:

Magnitude of directional derivative in search direction less than 2*options.TolFun and maximum constraint violation is less than options.TolCon

Active Constraints:

2

9

x =

0.00050378663220

0.00000000000000

30.00000000000000

feval =

-4.657237250542452e-035

Const. 1

Const. 2

Const. 3

Const. 5

Const. 4

Const. 6

Const. 7

Const. 8

Const. 9

Sequence: A,B,Aeq,Beq,LB,UB,C,Ceq

Example (Cont.)

Multiobjective Optimization currently solve this type of problem, switching to medium-scale (line search).

• Previous examples involved problems with a single

objective function.

• Now let us look at solving problem with multiobjective

function by lsqnonlin.

• Example is taken for data curve fitting

• In curve fitting problem the the error is reduced at each time step producing multiobjective function.

lsqnonlin in Matlab currently solve this type of problem, switching to medium-scale (line search). – Curve fitting

• clc; %recfit.m

• clear;

• global data;

• data= [ 0.6000 0.999

• 0.6500 0.998

• 0.7000 0.997

• 0.7500 0.995

• 0.8000 0.982

• 0.8500 0.975

• 0.9000 0.932

• 0.9500 0.862

• 1.0000 0.714

• 1.0500 0.520

• 1.1000 0.287

• 1.1500 0.134

• 1.2000 0.0623

• 1.2500 0.0245

• 1.3000 0.0100

• 1.3500 0.0040

• 1.4000 0.0015

• 1.4500 0.0007

• 1.5000 0.0003 ]; % experimental data,`1st coloum x, 2nd coloum R

• x=data(:,1);

• Rexp=data(:,2);

• plot(x,Rexp,'ro'); % plot the experimental data

• hold on

• b0=[1.0 1.0]; % start values for the parameters

• b=lsqnonlin('recfun',b0) % run the lsqnonlin with start value b0, returned parameter values stored in b

• Rcal=1./(1+exp(1.0986/b(1)*(x-b(2)))); % calculate the fitted value with parameter b plot(x,Rcal,'b'); % plot the fitted value on the same graph

Find b1 and b2

>>recfit

>>b =0.0603 1.0513

%recfun.m

function y=recfun(b)

global data;

x=data(:,1);

Rexp=data(:,2);

Rcal=1./(1+exp(1.0986/b(1)*(x-b(2))));

% the calculated value from the model

%y=sum((Rcal-Rexp).^2);

y=Rcal-Rexp;

% the sum of the square of the difference

%between calculated value and experimental value

• Short tutorial Model Fitting last edited on 26 October 2003 at 7:22 pm by westlake.che.gatech.edu

Simulink Example currently solve this type of problem, switching to medium-scale (line search).

Shooting a flying box

Eq. of ball motion in z horz. direction

Eq. of ball motion in h vert. direction

Aerodynamic drag force

Angle of ball

Simulink example – shooting ball currently solve this type of problem, switching to medium-scale (line search).

InitialGuess= pi/2.5 ;

X = fminsearch('Distflysim', InitialGuess)*180/pi;

fprintf('\nShoot at %f deg \n', X);

function P = Distflysim(theta_0)

F0=25.0; %N

cart_mass=2; %kg

x_dot_max=50; %m/sec

ro_air=1.224; %kg/m^3

h0=0.5; %m

z0=0;

Cd=1;

r_ball=0.05; %m

A_ball=pi*r_ball^2;

ball_mass=0.1; %kg

g=-9.8; %m/sec^2

V0=50; %m/secF0=15.0; %N

AeroFac=Cd*A_ball*ro_air/2;

theta_0

assignin('base','F0',F0);

assignin('base','cart_mass',cart_mass);

assignin('base','x_dot_max',x_dot_max);

assignin('base','AeroFac',AeroFac);

assignin('base','ball_mass',ball_mass);

assignin('base','g',g);

assignin('base','V0',V0);

assignin('base','theta_0',theta_0);

% save ShotParams.mat Newrtp;

np=max(size(y));

xf=y(np,1);

zf=y(np,2);

%hf=y(np,3);

P=(xf-zf)^2;%+(hf-25)^2;

F0=25.0; %N

cart_mass=1; %kg

x_dot_max=50; %m/sec

ro_air=1.224; %kg/m^3

Cd=1;

r_ball=0.05; %m

A_ball=pi*r_ball^2;

ball_mass=0.05; %kg

g=-9.8; %m/sec^2