10701 Recitation 5 Duality and SVM

1. 10701 Recitation 5Duality and SVM Ahmed Hefny

2. Outline • Langrangian and Duality • The Lagrangian • Duality • Examples • Support Vector Machines • Primal Formulation • Dual Formulation • Soft Margin and Hinge Loss

3. Lagrangian • Consider the problem s.t. • Add a Lagrange multiplier for each constraint

4. Lagrangian • Lagrangian • Setting gradient to 0 gives • [Feasible point] [Cannot decrease except by violating constraints]

5. Lagrangian • Consider the problem s.t. • Add a Lagrange multiplier for each constraint

6. Duality

7. Duality • Primal problem s.t. • Equivalent to

8. Duality • Primal problem s.t. • Equivalent to

9. Duality • Dual Problem • Dual function: • Concave, regardless of the convexity of the primal • Lower bound on primal Lagrangian Dual Function

10. Duality Primal Problem

11. Duality Primal Problem • For each row (choice of ), • pick the largest element • then select the minimum.

12. Duality Dual Problem • For each column (choice of ), • pick the smallest element • then select the maximum.

13. Duality Claim:

14. Duality Claim: For any The difference between primal minimum And dual maximum is called duality gap duality gap = 0  Strong Duality

15. Duality When does

16. Duality When does is a saddle point

17. Duality When does is a saddle point Necessity  By definition of dual Sufficiency 

18. Duality When does is a saddle point Necessity  By definition of dual Sufficiency  The dual at is the upper bound

19. Duality • If strong duality holds, KKT conditions apply to optimal point • Stationary Point • Primal Feasibility • Dual Feasibility () • Complementary Slackness () • KKT conditions are • Sufficient • Necessary under strong duality

20. Example: LP • Primal s.t.

21. Example: LP • Primal s.t. • Lagrangian

22. Example: LP • Dual Function

23. Example: LP • Dual Function • Set gradient w.r.t to 0

24. Example: LP • Dual Function • Set gradient w.r.t to 0 • Dual Problem s.t. Why keep this as a constraint ?

25. Example: LASSO • We will use duality to transform LASSO into a QP

26. Example: LASSO Primal What is the dual function in this case ?

27. Example: LASSO Reformulated Primal s.t. Dual

28. Example: LASSO Dual Setting gradient to zero gives

29. Example: LASSO • Dual Problem s.t.

30. Support Vector Machines docs.opencv.org

31. Support Vector Machines • Find the maximum margin hyper-plane • “Distance” from a point to the hyper-plane is given by • Max Margin:

32. Support Vector Machines • Max Margin • Unpleasant (max min ?) • No Unique Solution

33. Support Vector Machines • Max Margin s.t. ???

34. Support Vector Machines • Max Margin s.t.

35. Support Vector Machines • Max Margin s.t.

36. Support Vector Machines • Max Margin (Canonical Representation) s.t. • QP, much better than

37. SVM Dual Problem Recall that the Lagrangian is formed by adding a Lagrange multiplier for each constraint.

38. SVM Dual Problem Fix and minimize w.r.t :

39. SVM Dual Problem Fix and minimize w.r.t : Plug-in Constraint (why ?)

40. SVM Dual Problem Dual Problem s.t. Another QP. So what ?

41. SVM Dual Problem • Only Inner products  Kernel Trick • Complementary Slackness  Support Vectors • KKT conditions lead to Efficient optimization algorithms (compared to general QP solver)

42. SVM Dual Problem • Classification of a test point • To get use the fact that for any support vector. • For numerical stability, average over all support vectors.

43. Soft Margin SVM Hard Margin SVM , where

44. Soft Margin SVM Hard Margin SVM , where loss regularization

45. Soft Margin SVM Relax it a little bit , where

46. Soft Margin SVM Relax it a little bit , where

47. Soft Margin SVM Relax it a little bit

48. Soft Margin SVM Equivalent Formulation s.t.

49. Conclusions • Duality allows for establishing a lower bound on minimization problem. • Key idea • “min max” upper bounds “max min” • Strong Duality  Necessity of KKT Conditions • Duality on SVMs • Kernel Trick • Support Vectors • Soft Margin SVM = Hinge Loss

50. Resources • Bishop, “Pattern Recognition and Machine Learning”, Chp 7 • Gordon & Tibshirani, 10725 Optimization (Fall 2012) Lecture Slides: http://www.cs.cmu.edu/~ggordon/10725-F12/schedule.html • Fiterau, Kernels and SVM “http://alex.smola.org/teaching/cmu2013-10-701/slides/6_Recitation_Kernels.pdf”