NLP
This presentation is the property of its rightful owner.
Sponsored Links
1 / 58

NLP PowerPoint PPT Presentation


  • 135 Views
  • Uploaded on
  • Presentation posted in: General

NLP. KKT Practice and Second Order Conditions from Nash and Sofer. Unconstrained. First Order Necessary Condition Second Order Necessary Second Order Sufficient. Easiest Problem. Linear equality constraints. KKT Conditions. Note for equality – multipliers are unconstrained

Download Presentation

NLP

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Nlp

NLP

KKT Practice and Second Order Conditions from Nash and Sofer


Unconstrained

Unconstrained

  • First Order Necessary Condition

  • Second Order Necessary

  • Second Order Sufficient


Easiest problem

Easiest Problem

  • Linear equality constraints


Kkt conditions

KKT Conditions

Note for equality – multipliers are unconstrained

Complementarity not an issue


Null space representation

Null Space Representation

  • Let x* be a feasible point, Ax*=b.

  • Any other feasible point can be written as x=x*+p where Ap=0

  • The feasible region

    {x : x*+p pN(A)}

    where N(A) is null space of A


Null and range spaces

Null and Range Spaces

See Section 3.2 of Nash and Sofer for example


Orthogonality

Orthogonality


Null space review

Null Space Review


Constrained to unconstrained

Constrained to Unconstrained

You can convert any linear equality constrained optimization problem to an equivalent unconstrained problem

  • Method 1 substitution

  • Method 2 using Null space representation and a feasible point.


Example

Example

  • Solve by substitution

    becomes


Null space method

Null Space Method

  • x*= [4 0 0]’

  • x=x*+Zv

    becomes


General method

General Method

  • There exists a Null Space Matrix

  • The feasible region is:

  • Equivalent “Reduced” Problem


Optimality conditions

Optimality Conditions

  • Assume feasible point and convert to null space formulation


Where is kkt

Where is KKT?

  • KKT implies null space

  • Null Space implies KKT

    Gradient is not in Null(A), thus it must be in Range(A’)


Lemma 14 1 necessary conditions

Lemma 14.1 Necessary Conditions

  • If x* is a local min of f over {x|Ax=b}, and Z is a null matrix

  • Or equivalently use KKT Conditions


Lemma 14 2 sufficient conditions

Lemma 14.2 Sufficient Conditions

  • If x* satisfies (where Z is a basis matrix for Null(A))

    then x* is a strict local minimizer


Lemma 14 2 sufficient conditions kkt form

Lemma 14.2 Sufficient Conditions (KKT form)

  • If (x*,*) satisfies (where Z is a basis matrix for Null(A))

    then x* is a strict local minimizer


Lagrangian multiplier

Lagrangian Multiplier

  • * is called the Lagrangian Multiplier

  • It represents the sensitivity of solution to small perturbations of constraints


Optimality conditions1

Optimality conditions

  • Consider min (x2+4y2)/2 s.t. x-y=10


Optimality conditions2

Optimality conditions

  • Find KKT point Check SOSC


In class practice

In Class Practice

  • Find a KKT point

  • Verify SONC and

    SOSC


Linear equality constraints i

Linear Equality Constraints - I


Linear equality constraints ii

Linear Equality Constraints - II


Linear equality constraints iii

Linear Equality Constraints - III

so SOSC satisfied, and x* is a strict local minimum

Objective is convex, so KKT conditions are sufficient.


Next easiest problem

Next Easiest Problem

  • Linear equality constraints

    Constraints form a polyhedron


Close to equality case

x*

Close to Equality Case

Equality FONC:

a2x = b

a2x = b

-a2

Polyhedron Ax>=b

a3x = b

a4x = b

-a1

a1x = b

contour set of function

unconstrained minimum

Which i are 0? What is the sign of I?


Close to equality case1

x*

Close to Equality Case

Equality FONC:

a2x = b

a2x = b

-a2

Polyhedron Ax>=b

a3x = b

a4x = b

-a1

a1x = b

Which i are 0? What is the sign of I?


Inequality case

x*

Inequality Case

Inequality FONC:

a2x = b

a2x = b

-a2

Polyhedron Ax>=b

a3x = b

a4x = b

-a1

a1x = b

Nonnegative Multipliers imply gradient points to the less than

Side of the constraint.


Lagrangian multipliers

Lagrangian Multipliers


Lemma 14 3 necessary conditions

Lemma 14.3 Necessary Conditions

  • If x* is a local min of f over {x|Ax≤b}, and Z is a null-space matrix for active constraints then for some vector *


Lemma 14 5 sufficient conditions kkt form

Lemma 14.5 Sufficient Conditions (KKT form)

  • If (x*,*) satisfies


Lemma 14 5 sufficient conditions kkt form1

Lemma 14.5 Sufficient Conditions (KKT form)

where Z+ is a basis matrix for Null(A +) and A + corresponds to nondegenerate active constraints)

i.e.


Sufficient example

Sufficient Example

  • Find solution and verify SOSC


Linear inequality constraints i

Linear Inequality Constraints - I


Linear inequality constraints ii

Linear Inequality Constraints - II


Linear inequality constraints iii

Linear Inequality Constraints - III


Linear inequality constraints iv

Linear Inequality Constraints - IV


Example1

Example

  • Problem


You try

You Try

  • Solve the problem using above theorems:


Why necessary and sufficient

Why Necessary and Sufficient?

  • Sufficient conditions are good for?

    • Way to confirm that a candidate point

    • is a minimum (local)

    • But…not every min satisifies any given SC

  • Necessary tells you:

    • If necessary conditions don’t hold then you know you don’t have a minimum.

    • Under appropriate assumptions, every point that is a min satisfies the necessary cond.

    • Good stopping criteria

    • Algorithms look for points that satisfy Necessary conditions


General constraints

General Constraints


Lagrangian function

Lagrangian Function

  • Optimality conditions expressed using

    Lagrangian function

    and Jacobian matrix

    were each row is a gradient of a constraint


Theorem 14 2 sufficient conditions equality kkt form

Theorem 14.2 Sufficient Conditions Equality (KKT form)

  • If (x*,*) satisfies


Theorem 14 4 sufficient conditions inequality kkt

Theorem 14.4 Sufficient Conditions Inequality (KKT)

  • If (x*,*) satisfies


Lemma 14 4 sufficient conditions kkt form

Lemma 14.4 Sufficient Conditions (KKT form)

where Z+ is a basis matrix for Null(A +) and A + corresponds to Jacobian of nondegenerate active constraints)

i.e.


Sufficient example1

Sufficient Example

  • Find solution and verify SOSC


Nonlinear inequality constraints i

Nonlinear Inequality Constraints - I


Nonlinear inequality constraints ii

Nonlinear Inequality Constraints - II


Nonlinear inequality constraints iii

Nonlinear Inequality Constraints - III


Sufficient example2

Sufficient Example

  • Find solution and verify SOSC


Nonlinear inequality constraints v

Nonlinear Inequality Constraints - V


Nonlinear inequality constraints vi

Nonlinear Inequality Constraints - VI


Theorem 14 1 necessary conditions equality

Theorem 14.1 Necessary Conditions- Equality

  • If x* is a local min of f over {x|g(x)=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then


Theorem 14 3 necessary conditions

Theorem 14.3 Necessary Conditions

  • If x* is a local min of f over {x|g(x)>=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then


Regular point

Regular point

  • If x* is a regular point with respect to the constraints g(x*) if the gradient of the active constraints are linearly independent.

  • For equality constraints, all constraints are active so

    should have linearly independent rows.


Necessary example

Necessary Example

  • Show optimal solution x*=[1,0]’

    is regular and find KKT point


Constraint qualifications

Constraint Qualifications

  • Regularity is an example of a constraint qualification CQ.

  • The KKT conditions are based on linearizations of the constraints.

  • CQ guarantees that this linearization is not getting us into trouble. Problem is

    KKT point might not exist.

  • There are many other CQ,e.g., for inequalities Slater is there exists g(x)<0.

  • Note CQ not needed for linear constraints.


Kkt summary

KKT Summary

X* is global min

Convex f

Convex constraints

X* is local min

SOSC

CQ

KKT Satisfied


  • Login