Model Selection for SVM & Our intent works. Songcan Chen Feb. 8, 2012. Outline. Model Selection for SVM Our intent works. Model Selection for SVM. Introduction to 2 works. Introduction to 2 works. Model selection for primal SVM [MBB11, MLJ11] Selection of Hypothesis Space

Download Presentation

Model Selection for SVM & Our intent works

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

Introduction to 2 works • Model selection for primal SVM[MBB11, MLJ11] • Selection of Hypothesis Space • Selecting the Hypothesis Space for Improving the Generalization Ability of Support Vector Machines[AGOR11,IJCNN2011] • The Impact of Unlabeled Patterns in Rademacher Complexity Theory for Kernel Classifiers [AGOR11,NIPS2011]

1st work • Model selection for primal SVM [MBB11, MLJ11] [MBB11] Gregory Moore · Charles Bergeron · Kristin P. Bennett, Machine Learning (2011) 85:175–208

Outline • Primal SVM • Model selection 1) Bilevel Program for CV 2) Two optimization Methods: Impilicit & Explicit methods 3) Experiments 4) Conclusions

Primal SVM • Advantages: 1) simple to implement, theoretically sound, and easy to customize to different tasks such as classification, regression, ranking and so forth. 2) very fast, linear in the number of samples • Difficulty model selection

Model selection An often-adopted approach: Cross-validation (CV) over a grid Advantage: simple and almost universal! Weakness: high computation exponential in the number of hyperparameters and the number of grid points for each hyperparameter.

Motivation • CV is naturally and precisely formulated as a bilevel program (BP) shown as follows. Bilevel CV Problem (BCP)

Bilevel CV Problem (BCP) (1) BCP for a single validation and training split: • The outer-level leader problem selects the hyperparameters, γ, to perform well on avalidation set. • The follower problem trains an optimal inner-level model for the given hyperparameters, and returns a weightvectorw for validation.

Bilevel CV Problem (BCP) (2) More Specifically, Model selection via T-fold CV BCP! • The inner-level problems minimize theregularized training error to determine the best function for the given hyperparameters for each fold. • The hyperparameters are the outer-level control variables. The objective of the outer-level is to minimize the validation error based on the optimal parameters (w) returned for each fold.

Formal Formulation for BCP (1) • Given a training sample Ω:= {xj, yj}, j=1… l∈ Rn+1. • T-CV: PartitionΩ into T equally sized divisions; then for fold t=1…T, one of the divisions is used as the validation set, , and the remaining T-1 divisions are assigned to the training set, . • Letγ∈Rmbe the set of m model hyperparameters and wtbe the model weights for the t-th fold.

Formal Formulation for BCP (2) Let be the inner-level training function given the t-th fold training dataset and be the t-th outer-level validation loss function given its validation dataset

Implicit method (1) i.e., make wan implicit function ofγ, namely w(γ). (4) Forming a nonlinear objective! In practice, nonlinear objectives are much easier to optimize than nonlinear constraints. Where w(γ)is computed such that

Implicit method (2) Since for optimality, Equivalently having the KKT condition (5)

Summary for ImpGrad • ImpGrad alternates between training a model and updating the hyperparameters. Ideally an explicit algorithm that simultaneously solves for both model weights and hyperparameters would be more efficient as there is no need to train a model to optimality when far from the optimal solution.

Explicit Methods (1) Assume that the inner-level objective functions are differentiable and convex with respect to w, thus the optimality condition is the partial derivative of Lin(w, γ)with respect to wis equal to zero:

Explicit Methods (2) In position to transforming a bilevel program to the nonconvex nonsmooth program

Summary • Coarse grid search was reasonably fast; faster than both ImpGrad and PBP. In terms of generalization though, coarse grid search performed the worst. • Implicit and PBP algorithms performed better, with PBP being faster on the smaller datasets and ImpGrad being faster on the larger datasets. Generalization was slightly better for PBP.

Conclusions (1) • ImpGradfinds solutions with good generalization very quickly for large datasets, but illustrates more erratic behavior on all of the small datasets. • PBPuses a well-founded subgradient method with proven convergence properties and yields a robust explicit algorithm that performed well on problems of all sizes. While it appears to be roughly linear in the training time required per modeling set size.

Conclusions (2) • Like all machine learning algorithms, PBP and ImpGrad have algorithm parameters that must be defined such as exit criteria, starting points, and proximity parameters. • ImpGrad and PBP assume that the inner-level objective functions are at least once differentiable.