1 / 60

Exploiting Duality (Particularly the dual of SVM)

VISUAL GEOMETRY GROUP. Exploiting Duality (Particularly the dual of SVM). M. Pawan Kumar. PART I : General duality theory. Basics of Mathematical Optimization The algebra The geometry Examples. PART II : Solving the SVM dual. General Decomposition Algorithm Good Working Set

livi
Download Presentation

Exploiting Duality (Particularly the dual of SVM)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. VISUAL GEOMETRY GROUP Exploiting Duality(Particularly the dual of SVM) M. Pawan Kumar

  2. PART I : General duality theory • Basics of Mathematical Optimization • The algebra • The geometry • Examples PART II : Solving the SVM dual • General Decomposition Algorithm • Good Working Set • Implementation Details

  3. Mathematical Optimization min f0(x) Objective function s.t. fi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints x is a feasible point fi(x) ≤ 0, hi(x) = 0 x is a strictlyfeasible point fi(x) < 0, hi(x) = 0 Feasible region - set of all feasible points

  4. Convex Optimization min f0(x) Objective function s.t. fi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints Feasible region is convex Objective function is convex Convex set??? Convex function???

  5. Convex Set Line Segment x1 x2 c x1 + (1 - c) x2 c  [0,1] Endpoints

  6. Convex Set x1 x2 All points on the line segment lie within the set For all line segments with endpoints in the set

  7. Non-Convex Set x1 x2

  8. Examples of Convex Sets x1 x2 Line Segment

  9. Examples of Convex Sets x1 x2 Line

  10. Examples of Convex Sets Hyperplane aTx - b = 0

  11. Examples of Convex Sets Halfspace aTx - b ≤ 0

  12. Examples of Convex Sets t x2 x1 Second-order Cone ||x|| ≤ t

  13. Operations that Preserve Convexity Intersection Polyhedron / Polytope

  14. Operations that Preserve Convexity Intersection

  15. Operations that Preserve Convexity Affine Transformation x  Ax + b

  16. Convex Function f(x) x1 x2 x Blue point always lies above red point

  17. Convex Function f(x) x1 x2 x f( c x1 + (1 - c) x2 ) ≤ c f(x1) + (1 - c) f(x2) Domain of f(.) has to be convex

  18. Convex Function f(x) x1 x2 x f( c x1 + (1 - c) x2 ) ≤ c f(x1) + (1 - c) f(x2) -f(.) is concave

  19. Convex Function Once-differentiable functions f(y) + f(y)T (x - y) ≤ f(x) f(x) (y,f(y)) f(y) + f(y)T (x - y) x Twice-differentiable functions 2f(x) 0

  20. Convex Function and Convex Sets f(x) x Epigraph of a convex function is a convex set

  21. Examples of Convex Functions Linear function aTx p-Norm functions (x1p + x2p + xnp)1/p, p ≥ 1 Quadratic functions xTQx Q 0

  22. Operations that Preserve Convexity Non-negative weighted sum f1(x) f2(x) + w2 + …. w1 x x xTQx + aTx + b Q 0

  23. Operations that Preserve Convexity Pointwise maximum f1(x) f2(x) , max x x Pointwise minimum of concave functions is concave

  24. Convex Optimization min f0(x) Objective function s.t. fi(x) ≤ 0 Inequality constraints hi(x) = 0 Equality constraints Feasible region is convex  Objective function is convex 

  25. PART I : General duality theory • Basics of Mathematical Optimization • The algebra • The geometry • Examples PART II : Solving the SVM dual • General Decomposition Algorithm • Good Working Set • Implementation Details

  26. Lagrangian min f0(x) s.t. fi(x) ≤ 0 hi(x) = 0 f0(x) L(x,,) + ∑i i fi(x) i ≥ 0 + ∑i i hi(x)

  27. Lagrangian Dual f0(x) L(x,,) + ∑i i fi(x) i ≥ 0 + ∑i i hi(x) g(,) minx L(x,,) x belongs to intersection of domains of f0, fi and hi x D

  28. Lagrangian Dual g(,) = f0(x) minx + ∑i i fi(x) i ≥ 0 + ∑i i hi(x) Pointwise minimum of affine (concave) functions Dual function is concave

  29. Lagrangian Dual p* = min f0(x) ≥ s.t. fi(x) ≤ 0 For all (,) hi(x) = 0 g(,) = f0(x) minx + ∑i i fi(x) i ≥ 0 + ∑i i hi(x)

  30. The Dual Problem The lower bound could be far from p* Best lower bound? Easy to obtain d* = max, f0(x) minx + ∑i i fi(x) i ≥ 0 + ∑i i hi(x) p* - d* ≥ 0 Duality Gap

  31. G u The Geometric Interpretation u v t (fi(x), hi(x), f0(x)) G x D t p*

  32. The Geometric Interpretation (, ,1)T (u, v, t) ≥ g(, ) t G p* d* u g()

  33. The Duality Gap p* = min f0(x) ≥ s.t. fi(x) ≤ 0 hi(x) = 0 d* = f0(x) + ∑i i fi(x) i ≥ 0 + ∑i i hi(x) max, minx

  34. The Duality Gap p* - d* Duality Gap p* - d* ≥ 0 Weak Duality p* - d* = 0 Strong Duality

  35. Strong Duality Problem is convex There exists a strictly feasible point Taken care of by most solvers Slater’s Condition

  36. At Strong Duality f0(x*) = g(*, *) = minx ( f0(x) + ∑i i*fi(x) + ∑ii*hi(x)) ≤ f0(x*) + ∑i i*fi(x*) + ∑ii*hi(x*) ≤ f0(x*) Inequalities hold with equality x* minimizes the Lagrangian at (*, *)

  37. At Strong Duality f0(x*) = g(*, *) = minx ( f0(x) + ∑i i*fi(x) + ∑ii*hi(x)) ≤ f0(x*) + ∑i i*fi(x*) + ∑ii*hi(x*) ≤ f0(x*) Inequalities hold with equality i*fi(x*) = 0

  38. KKT Conditions fi(x*) ≤ 0 hi(x*) = 0 Primal feasible i* ≥ 0 Dual feasible i*fi(x*) = 0 Complementary Slackness f0(x*) + ∑i i*fi(x*) + ∑i i*hi(x*) = 0 Necessary conditions for strong duality

  39. KKT Conditions fi(x*) ≤ 0 hi(x*) = 0 Primal feasible i* ≥ 0 Dual feasible i*fi(x*) = 0 Complementary Slackness f0(x*) + ∑i i*fi(x*) + ∑i i*hi(x*) = 0 Necessary and sufficient for convex problems

  40. PART I : General duality theory • Basics of Mathematical Optimization • The algebra • The geometry • Examples PART II : Solving the SVM dual • General Decomposition Algorithm • Good Working Set • Implementation Details

  41. Linear Program min cTx s.t. Ax = b x ≥ 0

  42. QCQP min (1/2)xTP0x + q0x + r0 s.t. (1/2)xTPix + qix + ri

  43. Entropy Maximization min ∑i xi log(xi) s.t. Ax ≤ b ∑i xi= 1

  44. 2/||w|| The SVM Framework wTx + b = 0 min 1/2 wTw + C  i yi (wTxi + b) ≥ 1 - i i ≥ 0 Points X = {xi} Convex Quadratic Program Labels y= {yi} yi {-1, +1}

  45. The SVM Dual min (1/2) TQ - T1 s.t. Ty = 0 0 ≤  ≤ C1 Qij = yiyjxiTxj = yiyj k(xi,xj)

  46. PART I : General duality theory • Basics of Mathematical Optimization • The algebra • The geometry • Examples PART II : Solving the SVM dual • General Decomposition Algorithm • Good Working Set • Implementation Details

  47. The SVM Dual min (1/2) TQ - T1 s.t. Ty = 0 0 ≤  ≤ C1 Choose ‘q’ variables. Fix the rest. Best set B? Change unfixed variables, satisfying constraints, to decrease objective function (small problem). Repeat. Minimum ‘q’ ??? Till When ???

  48. KKT Conditions min (1/2) TQ - T1 s.t. Ty = 0 0 ≤  ≤ C1 eq iup ilo g() -1 + Q + eqy - lo + up = 0 ilo i = 0 iup (i - C) = 0 ilo ≥ 0 iup ≥ 0

  49. KKT Conditions -1 + g() + eqy - lo + up = 0 ilo i = 0 iup (i - C) = 0 ilo ≥ 0 iup ≥ 0 For all 0 < i < C -1 + gi() + eqyi = 0 For all i = 0 -1 + gi() + eqyi - ilo = 0 For all i = C -1 + gi() + eqyi + iup = 0

  50. KKT Conditions -1 + g() + eqy - lo + up = 0 ilo i = 0 iup (i - C) = 0 ilo ≥ 0 iup ≥ 0 gi() = yi∑j jyj k(xi,xj) git() = gi(t-1) + yi∑j B (jt - jt-1)yj k(xi,xj) Best set of ‘q’ variables (Working set)

More Related