Trading Convexity for Scalability

1. Marco A. Alvarez CS7680 Department of Computer Science Utah State University Trading Convexity for Scalability

2. Paper • Collobert, R., Sinz, F., Weston, J., and Bottou, L. 2006. Trading convexity for scalability. In Proceedings of the 23rd International Conference on Machine Learning (Pittsburgh, Pennsylvania, June 25 - 29, 2006). ICML '06, vol. 148. ACM Press, New York, NY, 201-208.

3. Introduction • Previously in Machine Learning • Non-convex cost function in MLP • Difficult to optimize • Work efficiently • SVM are defined by a convex function • Easier optimization (algorithms) • Unique solution (we can write theorems) • Goal of the paper • Sometimes non-convexity has benefits • Faster == training and testing (less support vectors) • Non-convex SVMs (faster and sparser) • Fast transductive SVMs

4. From SVM • Decision function • Primal formulation • Minimize ||w|| so that margin is maximized • w is a combination of a small number of data (sparsity) • Decision boundary is determined by the support vectors • Dual formulation s.t.

5. SVM problem • Number of support vectors increases linearly with L • Cost attributed to one example (x,y): • From:

6. Ramp Loss Function • Given: Outliers Non SV

7. Concave-Convex Procedure (CCCP) • Given a cost function: • Decompose into a convex part and a concave part • Is guaranteed to decrease at each iteration

8. Using the Ramp Loss

9. CCCP for Ramp Loss

10. Results

11. Speedup

12. Time and Number of SVs

13. Transductive SVMs

14. Loss Function • Cost to be minimized:

15. Balancing Constraint • Necessary for TSVMs

16. Results

17. Training Time

18. Quadratic Fit