1 / 71

# Convex Programming - PowerPoint PPT Presentation

Convex Programming. Brookes Vision Reading Group. Huh?. What is convex ??? What is programming ??? What is convex programming ???. Huh?. What is convex ??? What is programming ??? What is convex programming ???. Convex Function. f(t x + (1-t) y) <= t f(x) + (1-t) f(y). Convex Function.

Related searches for Convex Programming

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about 'Convex Programming' - vartouhi

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Convex Programming

Brookes Vision Reading Group

• What is convex ???

• What is programming ???

• What is convex programming ???

• What is convex ???

• What is programming ???

• What is convex programming ???

f(t x + (1-t) y) <= t f(x) + (1-t) f(y)

Is a linear function convex ???

Region above a convex function is a convex set.

Is the set of all positive semidefinite matrices convex??

• What is convex ???

• What is programming ???

• What is convex programming ???

Example

Constraints

Programming

• Objective function to be minimized/maximized.

• Constraints to be satisfied.

Optimal solution

Vertices

Objective function

Feasible region

• What is convex ???

• What is programming ???

• What is convex programming ???

• Convex optimization function

• Convex feasible region

• Why is it so important ???

• Global optimum can be found in polynomial time.

• Many practical problems are convex

• Non-convex problems can be relaxed to convex ones.

• Convex optimization function

• Convex feasible region

• Examples ???

• Linear Programming

• Second Order Cone Programming

• What ???

• Semidefinite Programming

• All this sounds Greek and Latin !!!!

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

2 out of 3 is not bad !!!

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

• || u || < t

• u - vector of dimension ‘d-1’

• t - scalar

• Cone lies in ‘d’ dimensions

• Second Order Cone defines a convex set

• Example: Second Order Cone in 3D

x2 + y2 <= z2

ICE CREAM !!

x2 + y2 <= z2

Affine mapping of SOC

Second Order Cone Programming

Minimize fTx

Subject to || Ai x+ bi || <= ciT x + di

i = 1, … , L

Constraints are SOC of ni dimensions

Feasible regions are intersections of conic regions

• A more general convex problem than LP

• LP  SOCP

• Fast algorithms for finding global optimum

• LP - O(n3)

• SOCP - O(L1/2) iterations of O(n2∑ni)

• Many standard problems are SOCP-able

• Sum of norms

• Maximum of norms

• Problems with hyperbolic constraints

• Sum of norms

• Maximum of norms

• Problems with hyperbolic constraints

Minimize xT P0 x + 2 q0T x + r0

Subject to xT Pi x + 2 qiT x + ri

Pi >= 0

|| P01/2 x + P0-1/2 x ||2 + r0 -q0TP0-1p0

Minimize xT P0 x + 2 q0T x + r0

Subject to xT Pi x + 2 qiT x + ri

Minimize t

Subject to || P01/2 x + P0-1/2 x || < = t

|| P01/2 x + P0-1/2 x || < = (r0 -q0TP0-1p0)1/2

• Sum of norms

• Maximum of norms

• Problems with hyperbolic constraints

Minimize  || Fi x + gi ||

Minimize  ti

Subject to || Fi x + gi || <= ti

Special Case: L-1 norm minimization

• Sum of norms

• Maximum of norms

• Problems with hyperbolic constraints

Minimize max || Fi x + gi ||

Minimize t

Subject to || Fi x + gi || <= t

Special Case: L-inf norm minimization

• Sum of norms

• Maximum of norms

• Problems with hyperbolic constraints

x >= 0 , y >= 0

w2 <= xy

|| [2w; x-y] || <= x+y

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

Linear Constraints

Semidefinite Programming

Minimize C  X

Subject to Ai X = bi

X >= 0

Linear Programming on Semidefinite Matrices

• A more general convex problem than SOCP

• LP  SOCP  SDP

• Generality comes at a cost though

• SOCP - O(L1/2) iterations of O(n2∑ni)

• SDP - O((∑ni)1/2) iterations of O(n2∑ni2)

• Many standard problems are SDP-able

• Minimizing the maximum eigenvalue

• Class separation with ellipsoids

• Minimizing the maximum eigenvalue

• Class separation with ellipsoids

Matrix M(z)

To find vector z* such that max is minimized.

Let max(M(z)) <= n

max(M(z)-nI) <= 0

min(nI - M(z)) >= 0

nI - M(z) >= 0

Matrix M(z)

To find vector z* such that max is minimized.

Max -n

nI - M(z) >= 0

• Minimizing the maximum eigenvalue

• Class separation with ellipsoids

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

Minimize xTQ0x + 2q0Tx + r0

Subject to xTQix + 2qiTx + ri < = 0

Qi >= 0

=> Convex

Non-Convex Quadratic Programming Problem !!!

Redefine x in homogenous coordinates.

y = (1; x)

Non-Convex Problems

Minimize xTQ0x + 2q0Tx + r0

Subject to xTQix + 2qiTx + ri < = 0

Minimize yTM0y

Subject to yTMiy < = 0

Mi = [ ri qiT; qi Qi]

• Problem is NP-hard.

• Let’s relax the problem to make it convex.

• Pray !!!

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

No donut for you !!!

SDP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Minimize M0 Y

Subject to Mi  Y < = 0

Y = yyT

SDP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Minimize M0 Y

Subject to Mi  Y < = 0

Y >= 0

Nothing left to do ….

but Pray

Note that we have squared the number of variables.

• Graph: G=(V,E)

• Maximum-Cut

- xi = -1

- xi = +1

Example - Max Cut

• Graph: G=(V,E)

• Maximum-Cut

• Graph: G=(V,E)

• Maximum-Cut

Alright !!! So it’s an integer programming problem !!!

Doesn’t look like quadratic programming to me !!!

Max Cut as an IQP

Naah !! Let’s get it into the standard quadratic form.

Max Cut as an IQP

Naah !! Let’s get it into the standard quadratic form.

To the white board.

(You didn’t think I’ll prepare slides for this, did you??)

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

X - xxT >= 0

SOCP Relaxation

Minimize yTM0y

Subject to yTMiy < = 0

Remember

Y = [1 xT; x X]

Minimize M0 Y

Subject to Mi  Y < = 0

Y >= 0

Say you’re given C = { C1, C2, … Cn} such that Cj >= 0

Cj (X - xxT) >= 0

(Ux)T (Ux) <= Cj X

Wait .. Isn’t this a hyperbolic constraint

Therefore, it’s SOCP-able.

Minimize yTM0y

Subject to yTMiy < = 0

Minimize Q0  X + 2q0Tx + r0

Subject to Qi  X + 2qiTx + ri < = 0

Cj (X - xxT) >= 0

Cj C

If C is the infinite set of all semidefinite matrices

SOCP Relaxation = SDP Relaxation

If C is finite,

SOCP relaxation is ‘looser’ than SDP relaxation.

Then why SOCP relaxation ???

Efficiency - Accuracy Tradeoff

Remember we had squared the number of variables.

Let’s try to reduce them with our choice of C.

For a general problem - Kim and Kojima

Using the structure of a specific problem -

e.g. Muramatsu and Suzuki for Max Cut

Minimize cT x

Subject to Qi  X + 2qiTx + ri < = 0

Q X + 2qTx + r <= 0

Q = n i uiuiT

Let1 >= 2 >= …. k >=0 >= k+1 >=n

Q+ = k i uiuiT

C =

Q X + 2qTx + r <= 0

xT Q+ x - Q+ X <= 0

xT Q+ x + k+1i uiuiT X + 2qTx + r <= 0

zi

Q+ = k i uiuiT

C =

uiuiT i = k+1, k+2, … n

Q X + 2qTx + r <= 0

xT Q+ x + k+1i zi+ 2qTx + r <= 0

xTuiuiTx - uiuiT X <= 0

Q+ = k i uiuiT

C =

uiuiT i = k+1, k+2, … n

Q X + 2qTx + r <= 0

xT Q+ x + k+1i zi+ 2qTx + r <= 0

xTuiuiTx - zi <= 0

ei eiT

i = 1, … , |V|

C =

uij uijT

(i,j)  E

vij vijT

(i,j)  E

Specific Problem Example - Max Cut

ei = [0 0 …. 1 0 …0]

uij = ei + ej

vij = ei - ej

Warning: Scary equations to follow.

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP

• Convex Optimization

• Second Order Cone Programming (SOCP)

• Semidefinite Programming (SDP)

• Non-convex optimization

• SDP relaxations

• SOCP relaxations

• Optimization Algorithms

• Interior Point Method for SOCP

• Interior Point Method for SDP