Non parametric learning
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

Non-Parametric Learning PowerPoint PPT Presentation


  • 63 Views
  • Uploaded on
  • Presentation posted in: General

Non-Parametric Learning. Prof. A.L. Yuille Stat 231. Fall 2004. Chp 4.1 – 4.3. Parametric versus Non-Parametric. Previous lectures on MLE learning assumed a functional form for the probability distribution. We now consider an alternative non-parametric method based on window function.

Download Presentation

Non-Parametric Learning

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Non parametric learning

Non-Parametric Learning

Prof. A.L. Yuille

Stat 231. Fall 2004.

Chp 4.1 – 4.3.


Parametric versus non parametric

Parametric versus Non-Parametric

  • Previous lectures on MLE learning assumed a functional form for the probability distribution.

  • We now consider an alternative non-parametric method based on window function.


Non parametric

Non-Parametric

  • It is hard to develop probability models for some data.

  • Example: estimate the distribution of annual rainfall in the U.S.A. Want to model p(x,y) – probability that a raindrop hits a position (x,y).

  • Problems: (i) multi-modal density is difficult for parametric models, (ii) difficult/impossible to collect enough data at each point (x,y).


Intuition

Intuition

  • Assume that the probability density is locally smooth.

  • Goal: estimate the class density model p(x) from data

  • Method 1: Windows based on points x in space.


Windows

Windows

  • For each point x, form a window centred at x with volume Count the number of samples that fall in the window.

  • Probability density is estimated as:


Non parametric1

Non-Parametric

  • Goal: to design a sequence of windows

    so that at each point x

  • (f(x) is the true density).

  • Conditions for window design:

  • increasing spatial resolution.

    (ii) many samples at each point

    (iii)


Two design methods

Two Design Methods

  • Parzen Window: Fix window size:

  • K-NN: Fix no. samples in window:


Parzen window

Parzen Window

  • Parzen window uses a window function

  • Example:

  • (i) Unit hypercube:

    and 0 otherwise.

  • (ii) Gaussian in d-dimensions.


Parzen windows

Parzen Windows

  • No. of samples in the hypercube is

  • Volume

  • The estimate of the distribution is:

  • More generally, the window interpolates the data.


Parzen window example

Parzen Window Example

  • Estimate a density with five modes using Gaussian windows at scales h=1,0.5, 0.2.


Convergence proof

Convergence Proof.

  • We will show that the Parzen window estimator converges to the true density at each point x with increasing number of samples.


Proof strategy

Proof Strategy.

  • Parzen distribution

    is a random variable which depends on the samples used to estimate it.

  • We have to take the expectation of the distribution with respect to the samples.

  • We show that the expected value of the Parzen distribution will be the true distribution. And the expected variance of the Parzen distribution will tend to 0 as no. samples gets large.


Convergence of the mean

Convergence of the Mean

  • Result follows.


Convergence of variance

Convergence of Variance

  • Variance:


Example of parzen window

Example of Parzen Window

  • Underlying density is Gaussian. Window volume decreases as


Example of parzen window1

Example of Parzen Window

  • Underlying Density is bi-modal.


Parzen window and interpolation

Parzen Window and Interpolation.

  • In practice, we do not have an infinite number of samples.

  • The choice of window shape is important. This effectively interpolates the data.

  • If the window shape fits the local structure of the density, then Parzen windows are effective.


  • Login