Kernels l.jpg
This presentation is the property of its rightful owner.
Sponsored Links
1 / 10

Kernels PowerPoint PPT Presentation


  • 85 Views
  • Uploaded on
  • Presentation posted in: General

Kernels. CMPUT 466/551 Nilanjan Ray. Agenda. Kernel functions in SVM: A quick recapitulation Kernels in regression Kernels in k -nearest neighbor classifier Kernel function: a deeper understanding A case study. Kernel Functions: SVM. The dual cost function:. The non-linear classifier

Download Presentation

Kernels

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Kernels l.jpg

Kernels

CMPUT 466/551

Nilanjan Ray


Agenda l.jpg

Agenda

  • Kernel functions in SVM: A quick recapitulation

  • Kernels in regression

  • Kernels in k-nearest neighbor classifier

  • Kernel function: a deeper understanding

  • A case study


Kernel functions svm l.jpg

Kernel Functions: SVM

The dual cost function:

The non-linear classifier

in dual variables:

The kernel function K is symmetric and positive (semi)definite by definition


Input space to feature space l.jpg

Input Space to Feature Space

Picture taken from:

Kernel methods for pattern analysis

By Shawe-Taylor and Cristianini


Input space to feature space example l.jpg

Input Space to Feature Space: Example


Kernel ridge regression l.jpg

Kernel Ridge Regression

Consider the regression problem: fit the function to N data points

Basis functions are non-linear in x

Form the cost function:

The solution is given by:

where,

Using the identity

Ex. Prove this identity

we have

We have defined:

Note that

is the kernel matrix

Finally the solution is given by

The basis functions h have disappeared!


Kernel k nearest neighbor classifier l.jpg

Kernel k-Nearest Neighbor Classifier

Consider the k-nn classification problem in the feature space.

Basis functions are typically non-linear in x

The Euclidean distance in the feature space can be written as follows:

Once again, the basis functions h have disappeared!

Note also that a kernel function essentially provides similarity between two

points in the input space (opposite of distance measure!)


The kernel architecture l.jpg

The Kernel Architecture

Picture taken from:

Learning with kernels

By Scholkopf and Smola


Inside kernels l.jpg

Inside Kernels

Picture taken from:

Learning with kernels


Inside kernels10 l.jpg

Inside Kernels…

Given a point x in the input space, the function k(., x) is essentially function

So, x is mapped into a function space (known as Reproducing kernel Hilbert space (RKHS)

When we measure similarity of two points x and y in the input space, we are actually

measuring the similarity between two functions k(., x) and k(., y) in RKHS. How is this

similarity defined in in RKHS? By a (defined) inner product in RKHS:

Reproducing

property

All the solutions so far we obtained has the form:

This means these solutions are functions in RKHS.

Functions in RKHS are nicer: they are smooth, they have finite-dimensional representation.

Good for computations and practical solutions

See “Learning with kernels” for more; Must read G. Wahba’s work to learn more on RKHS vis-à-vis M/C Learning.


  • Login