slide1
Download
Skip this Video
Download Presentation
NLP

Loading in 2 Seconds...

play fullscreen
1 / 58

NLP - PowerPoint PPT Presentation


  • 184 Views
  • Uploaded on

NLP. KKT Practice and Second Order Conditions from Nash and Sofer. Unconstrained. First Order Necessary Condition Second Order Necessary Second Order Sufficient. Easiest Problem. Linear equality constraints. KKT Conditions. Note for equality – multipliers are unconstrained

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' NLP' - eagan-flowers


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

NLP

KKT Practice and Second Order Conditions from Nash and Sofer

unconstrained
Unconstrained
  • First Order Necessary Condition
  • Second Order Necessary
  • Second Order Sufficient
easiest problem
Easiest Problem
  • Linear equality constraints
kkt conditions
KKT Conditions

Note for equality – multipliers are unconstrained

Complementarity not an issue

null space representation
Null Space Representation
  • Let x* be a feasible point, Ax*=b.
  • Any other feasible point can be written as x=x*+p where Ap=0
  • The feasible region

{x : x*+p pN(A)}

where N(A) is null space of A

null and range spaces
Null and Range Spaces

See Section 3.2 of Nash and Sofer for example

constrained to unconstrained
Constrained to Unconstrained

You can convert any linear equality constrained optimization problem to an equivalent unconstrained problem

  • Method 1 substitution
  • Method 2 using Null space representation and a feasible point.
example
Example
  • Solve by substitution

becomes

null space method
Null Space Method
  • x*= [4 0 0]’
  • x=x*+Zv

becomes

general method
General Method
  • There exists a Null Space Matrix
  • The feasible region is:
  • Equivalent “Reduced” Problem
optimality conditions
Optimality Conditions
  • Assume feasible point and convert to null space formulation
where is kkt
Where is KKT?
  • KKT implies null space
  • Null Space implies KKT

Gradient is not in Null(A), thus it must be in Range(A’)

lemma 14 1 necessary conditions
Lemma 14.1 Necessary Conditions
  • If x* is a local min of f over {x|Ax=b}, and Z is a null matrix
  • Or equivalently use KKT Conditions
lemma 14 2 sufficient conditions
Lemma 14.2 Sufficient Conditions
  • If x* satisfies (where Z is a basis matrix for Null(A))

then x* is a strict local minimizer

lemma 14 2 sufficient conditions kkt form
Lemma 14.2 Sufficient Conditions (KKT form)
  • If (x*,*) satisfies (where Z is a basis matrix for Null(A))

then x* is a strict local minimizer

lagrangian multiplier
Lagrangian Multiplier
  • * is called the Lagrangian Multiplier
  • It represents the sensitivity of solution to small perturbations of constraints
optimality conditions1
Optimality conditions
  • Consider min (x2+4y2)/2 s.t. x-y=10
optimality conditions2
Optimality conditions
  • Find KKT point Check SOSC
in class practice
In Class Practice
  • Find a KKT point
  • Verify SONC and

SOSC

linear equality constraints iii
Linear Equality Constraints - III

so SOSC satisfied, and x* is a strict local minimum

Objective is convex, so KKT conditions are sufficient.

next easiest problem
Next Easiest Problem
  • Linear equality constraints

Constraints form a polyhedron

close to equality case

x*

Close to Equality Case

Equality FONC:

a2x = b

a2x = b

-a2

Polyhedron Ax>=b

a3x = b

a4x = b

-a1

a1x = b

contour set of function

unconstrained minimum

Which i are 0? What is the sign of I?

close to equality case1

x*

Close to Equality Case

Equality FONC:

a2x = b

a2x = b

-a2

Polyhedron Ax>=b

a3x = b

a4x = b

-a1

a1x = b

Which i are 0? What is the sign of I?

inequality case

x*

Inequality Case

Inequality FONC:

a2x = b

a2x = b

-a2

Polyhedron Ax>=b

a3x = b

a4x = b

-a1

a1x = b

Nonnegative Multipliers imply gradient points to the less than

Side of the constraint.

lemma 14 3 necessary conditions
Lemma 14.3 Necessary Conditions
  • If x* is a local min of f over {x|Ax≤b}, and Z is a null-space matrix for active constraints then for some vector *
lemma 14 5 sufficient conditions kkt form1
Lemma 14.5 Sufficient Conditions (KKT form)

where Z+ is a basis matrix for Null(A +) and A + corresponds to nondegenerate active constraints)

i.e.

sufficient example
Sufficient Example
  • Find solution and verify SOSC
example1
Example
  • Problem
you try
You Try
  • Solve the problem using above theorems:
why necessary and sufficient
Why Necessary and Sufficient?
  • Sufficient conditions are good for?
    • Way to confirm that a candidate point
    • is a minimum (local)
    • But…not every min satisifies any given SC
  • Necessary tells you:
    • If necessary conditions don’t hold then you know you don’t have a minimum.
    • Under appropriate assumptions, every point that is a min satisfies the necessary cond.
    • Good stopping criteria
    • Algorithms look for points that satisfy Necessary conditions
lagrangian function
Lagrangian Function
  • Optimality conditions expressed using

Lagrangian function

and Jacobian matrix

were each row is a gradient of a constraint

lemma 14 4 sufficient conditions kkt form
Lemma 14.4 Sufficient Conditions (KKT form)

where Z+ is a basis matrix for Null(A +) and A + corresponds to Jacobian of nondegenerate active constraints)

i.e.

sufficient example1
Sufficient Example
  • Find solution and verify SOSC
sufficient example2
Sufficient Example
  • Find solution and verify SOSC
theorem 14 1 necessary conditions equality
Theorem 14.1 Necessary Conditions- Equality
  • If x* is a local min of f over {x|g(x)=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then
theorem 14 3 necessary conditions
Theorem 14.3 Necessary Conditions
  • If x* is a local min of f over {x|g(x)>=0}, Z is a null-space matrix of the Jacobian g(x*)’, and x* is a regular point then
regular point
Regular point
  • If x* is a regular point with respect to the constraints g(x*) if the gradient of the active constraints are linearly independent.
  • For equality constraints, all constraints are active so

should have linearly independent rows.

necessary example
Necessary Example
  • Show optimal solution x*=[1,0]’

is regular and find KKT point

constraint qualifications
Constraint Qualifications
  • Regularity is an example of a constraint qualification CQ.
  • The KKT conditions are based on linearizations of the constraints.
  • CQ guarantees that this linearization is not getting us into trouble. Problem is

KKT point might not exist.

  • There are many other CQ,e.g., for inequalities Slater is there exists g(x)<0.
  • Note CQ not needed for linear constraints.
kkt summary
KKT Summary

X* is global min

Convex f

Convex constraints

X* is local min

SOSC

CQ

KKT Satisfied

ad