Convex programming
This presentation is the property of its rightful owner.
Sponsored Links
1 / 71

Convex Programming PowerPoint PPT Presentation


  • 45 Views
  • Uploaded on
  • Presentation posted in: General

Convex Programming. Brookes Vision Reading Group. Huh?. What is convex ??? What is programming ??? What is convex programming ???. Huh?. What is convex ??? What is programming ??? What is convex programming ???. Convex Function. f(t x + (1-t) y) <= t f(x) + (1-t) f(y). Convex Function.

Download Presentation

Convex Programming

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Convex programming

Convex Programming

Brookes Vision Reading Group


Convex programming

Huh?

  • What is convex ???

  • What is programming ???

  • What is convex programming ???


Convex programming

Huh?

  • What is convex ???

  • What is programming ???

  • What is convex programming ???


Convex function

Convex Function

f(t x + (1-t) y) <= t f(x) + (1-t) f(y)


Convex function1

Convex Function

Is a linear function convex ???


Convex set

Convex Set

Region above a convex function is a convex set.


Convex set1

Convex Set

Is the set of all positive semidefinite matrices convex??


Convex programming

Huh?

  • What is convex ???

  • What is programming ???

  • What is convex programming ???


Programming

Objective function

Example

Constraints

Programming

  • Objective function to be minimized/maximized.

  • Constraints to be satisfied.


Example

Example

Optimal solution

Vertices

Objective function

Feasible region


Convex programming

Huh?

  • What is convex ???

  • What is programming ???

  • What is convex programming ???


Convex programming1

Convex Programming

  • Convex optimization function

  • Convex feasible region

  • Why is it so important ???

  • Global optimum can be found in polynomial time.

  • Many practical problems are convex

  • Non-convex problems can be relaxed to convex ones.


Convex programming2

Convex Programming

  • Convex optimization function

  • Convex feasible region

  • Examples ???

  • Linear Programming

    • Refer to Vladimir/Pushmeet’s reading group

  • Second Order Cone Programming

    • What ???

  • Semidefinite Programming

    • All this sounds Greek and Latin !!!!


Outline

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP

2 out of 3 is not bad !!!


Outline1

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Second order cone

Second Order Cone

  • || u || < t

    • u - vector of dimension ‘d-1’

    • t - scalar

    • Cone lies in ‘d’ dimensions

  • Second Order Cone defines a convex set

  • Example: Second Order Cone in 3D

x2 + y2 <= z2


X 2 y 2 z 2

Hmmm

ICE CREAM !!

x2 + y2 <= z2


Second order cone programming

Linear Objective Function

Affine mapping of SOC

Second Order Cone Programming

Minimize fTx

Subject to || Ai x+ bi || <= ciT x + di

i = 1, … , L

Constraints are SOC of ni dimensions

Feasible regions are intersections of conic regions


Example1

Example


Why socp

Why SOCP ??

  • A more general convex problem than LP

    • LP  SOCP

  • Fast algorithms for finding global optimum

    • LP - O(n3)

    • SOCP - O(L1/2) iterations of O(n2∑ni)

  • Many standard problems are SOCP-able


Socp able problems

SOCP-able Problems

  • Convex quadratically constrained quadratic programming

  • Sum of norms

  • Maximum of norms

  • Problems with hyperbolic constraints


Socp able problems1

SOCP-able Problems

  • Convex quadratically constrained quadratic programming

  • Sum of norms

  • Maximum of norms

  • Problems with hyperbolic constraints


Convex programming

QCQP

Minimize xT P0 x + 2 q0T x + r0

Subject to xT Pi x + 2 qiT x + ri

Pi >= 0

|| P01/2 x + P0-1/2 x ||2 + r0 -q0TP0-1p0


Convex programming

QCQP

Minimize xT P0 x + 2 q0T x + r0

Subject to xT Pi x + 2 qiT x + ri

Minimize t

Subject to || P01/2 x + P0-1/2 x || < = t

|| P01/2 x + P0-1/2 x || < = (r0 -q0TP0-1p0)1/2


Socp able problems2

SOCP-able Problems

  • Convex quadratically constrained quadratic programming

  • Sum of norms

  • Maximum of norms

  • Problems with hyperbolic constraints


Sum of norms

Sum of Norms

Minimize  || Fi x + gi ||

Minimize  ti

Subject to || Fi x + gi || <= ti

Special Case: L-1 norm minimization


Socp able problems3

SOCP-able Problems

  • Convex quadratically constrained quadratic programming

  • Sum of norms

  • Maximum of norms

  • Problems with hyperbolic constraints


Maximum of norms

Maximum of Norms

Minimize max || Fi x + gi ||

Minimize t

Subject to || Fi x + gi || <= t

Special Case: L-inf norm minimization


You weren t expecting a question were you

You weren’t expecting a question, were you ??


Socp able problems4

SOCP-able Problems

  • Convex quadratically constrained quadratic programming

  • Sum of norms

  • Maximum of norms

  • Problems with hyperbolic constraints


Hyperbolic constraints

Hyperbolic Constraints

x >= 0 , y >= 0

w2 <= xy

|| [2w; x-y] || <= x+y


Let s see if everyone was awake

Let’s see if everyone was awake !


Outline2

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Semidefinite programming

Linear Objective Function

Linear Constraints

Semidefinite Programming

Minimize C  X

Subject to Ai X = bi

X >= 0

Linear Programming on Semidefinite Matrices


Why sdp

Why SDP ??

  • A more general convex problem than SOCP

    • LP  SOCP  SDP

  • Generality comes at a cost though

    • SOCP - O(L1/2) iterations of O(n2∑ni)

    • SDP - O((∑ni)1/2) iterations of O(n2∑ni2)

  • Many standard problems are SDP-able


Sdp able problems

SDP-able Problems

  • Minimizing the maximum eigenvalue

  • Class separation with ellipsoids


Sdp able problems1

SDP-able Problems

  • Minimizing the maximum eigenvalue

  • Class separation with ellipsoids


Minimizing the maximum eigenvalue

Minimizing the Maximum Eigenvalue

Matrix M(z)

To find vector z* such that max is minimized.

Let max(M(z)) <= n

max(M(z)-nI) <= 0

min(nI - M(z)) >= 0

nI - M(z) >= 0


Minimizing the maximum eigenvalue1

Minimizing the Maximum Eigenvalue

Matrix M(z)

To find vector z* such that max is minimized.

Max -n

nI - M(z) >= 0


Sdp able problems2

SDP-able Problems

  • Minimizing the maximum eigenvalue

  • Class separation with ellipsoids


Outline3

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Non convex problems

Non-Convex Problems

Minimize xTQ0x + 2q0Tx + r0

Subject to xTQix + 2qiTx + ri < = 0

Qi >= 0

=> Convex

Non-Convex Quadratic Programming Problem !!!

Redefine x in homogenous coordinates.

y = (1; x)


Non convex problems1

Let’s solve this now !!!

Non-Convex Problems

Minimize xTQ0x + 2q0Tx + r0

Subject to xTQix + 2qiTx + ri < = 0

Minimize yTM0y

Subject to yTMiy < = 0

Mi = [ ri qiT; qi Qi]


Non convex problems2

Non-Convex Problems

  • Problem is NP-hard.

  • Let’s relax the problem to make it convex.

  • Pray !!!


Outline4

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Sdp relaxation

Bad Constraint !!!!

No donut for you !!!

SDP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Minimize M0 Y

Subject to Mi  Y < = 0

Y = yyT


Sdp relaxation1

SDP Problem

SDP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Minimize M0 Y

Subject to Mi  Y < = 0

Y >= 0

Nothing left to do ….

but Pray

Note that we have squared the number of variables.


Example max cut

Example - Max Cut

  • Graph: G=(V,E)

  • Maximum-Cut


Example max cut1

- xi = -1

- xi = +1

Example - Max Cut

  • Graph: G=(V,E)

  • Maximum-Cut


Example max cut2

Example - Max Cut

  • Graph: G=(V,E)

  • Maximum-Cut

Alright !!! So it’s an integer programming problem !!!

Doesn’t look like quadratic programming to me !!!


Max cut as an iqp

Max Cut problem can be written as

Max Cut as an IQP

Naah !! Let’s get it into the standard quadratic form.


Max cut as an iqp1

Max Cut problem can be written as

Max Cut as an IQP

Naah !! Let’s get it into the standard quadratic form.


Solving max cut using sdp relaxations

Solving Max Cut using SDP Relaxations

To the white board.

(You didn’t think I’ll prepare slides for this, did you??)


Outline5

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Socp relaxation

X - xxT >= 0

SOCP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Remember

Y = [1 xT; x X]

Minimize M0 Y

Subject to Mi  Y < = 0

Y >= 0


Socp relaxation1

SOCP Relaxation

Say you’re given C = { C1, C2, … Cn} such that Cj >= 0

Cj (X - xxT) >= 0

(Ux)T (Ux) <= Cj X

Wait .. Isn’t this a hyperbolic constraint

Therefore, it’s SOCP-able.


Socp relaxation2

SOCP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Minimize Q0  X + 2q0Tx + r0

Subject to Qi  X + 2qiTx + ri < = 0

Cj (X - xxT) >= 0

Cj C


Socp relaxation3

SOCP Relaxation

If C is the infinite set of all semidefinite matrices

SOCP Relaxation = SDP Relaxation

If C is finite,

SOCP relaxation is ‘looser’ than SDP relaxation.

Then why SOCP relaxation ???

Efficiency - Accuracy Tradeoff


Choice of c

Choice of C

Remember we had squared the number of variables.

Let’s try to reduce them with our choice of C.

For a general problem - Kim and Kojima

Using the structure of a specific problem -

e.g. Muramatsu and Suzuki for Max Cut


Choice of c1

Choice of C

Minimize cT x

Subject to Qi  X + 2qiTx + ri < = 0

Q X + 2qTx + r <= 0

Q = n i uiuiT

Let1 >= 2 >= …. k >=0 >= k+1 >=n


Choice of c2

Choice of C

Q+ = k i uiuiT

C =

Q X + 2qTx + r <= 0

xT Q+ x - Q+ X <= 0

xT Q+ x + k+1i uiuiT X + 2qTx + r <= 0

zi


Choice of c3

Choice of C

Q+ = k i uiuiT

C =

uiuiT i = k+1, k+2, … n

Q X + 2qTx + r <= 0

xT Q+ x + k+1i zi+ 2qTx + r <= 0

xTuiuiTx - uiuiT X <= 0


Choice of c4

Choice of C

Q+ = k i uiuiT

C =

uiuiT i = k+1, k+2, … n

Q X + 2qTx + r <= 0

xT Q+ x + k+1i zi+ 2qTx + r <= 0

xTuiuiTx - zi <= 0


Specific problem example max cut

ei eiT

i = 1, … , |V|

C =

uij uijT

(i,j)  E

vij vijT

(i,j)  E

Specific Problem Example - Max Cut

ei = [0 0 …. 1 0 …0]

uij = ei + ej

vij = ei - ej


Specific problem example max cut1

Specific Problem Example - Max Cut

Warning: Scary equations to follow.


Outline6

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Outline7

Outline

  • Convex Optimization

    • Second Order Cone Programming (SOCP)

    • Semidefinite Programming (SDP)

  • Non-convex optimization

    • SDP relaxations

    • SOCP relaxations

  • Optimization Algorithms

    • Interior Point Method for SOCP

    • Interior Point Method for SDP


Back to work now

Back to work now !!!


  • Login