- 81 Views
- Uploaded on
- Presentation posted in: General

Non-Parametric Learning

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Non-Parametric Learning

Prof. A.L. Yuille

Stat 231. Fall 2004.

Chp 4.1 – 4.3.

- Previous lectures on MLE learning assumed a functional form for the probability distribution.
- We now consider an alternative non-parametric method based on window function.

- It is hard to develop probability models for some data.
- Example: estimate the distribution of annual rainfall in the U.S.A. Want to model p(x,y) – probability that a raindrop hits a position (x,y).
- Problems: (i) multi-modal density is difficult for parametric models, (ii) difficult/impossible to collect enough data at each point (x,y).

- Assume that the probability density is locally smooth.
- Goal: estimate the class density model p(x) from data
- Method 1: Windows based on points x in space.

- For each point x, form a window centred at x with volume Count the number of samples that fall in the window.
- Probability density is estimated as:

- Goal: to design a sequence of windows
so that at each point x

- (f(x) is the true density).
- Conditions for window design:
- increasing spatial resolution.
(ii) many samples at each point

(iii)

- Parzen Window: Fix window size:
- K-NN: Fix no. samples in window:

- Parzen window uses a window function
- Example:
- (i) Unit hypercube:
and 0 otherwise.

- (ii) Gaussian in d-dimensions.

- No. of samples in the hypercube is
- Volume
- The estimate of the distribution is:
- More generally, the window interpolates the data.

- Estimate a density with five modes using Gaussian windows at scales h=1,0.5, 0.2.

- We will show that the Parzen window estimator converges to the true density at each point x with increasing number of samples.

- Parzen distribution
is a random variable which depends on the samples used to estimate it.

- We have to take the expectation of the distribution with respect to the samples.
- We show that the expected value of the Parzen distribution will be the true distribution. And the expected variance of the Parzen distribution will tend to 0 as no. samples gets large.

- Result follows.

- Variance:

- Underlying density is Gaussian. Window volume decreases as

- Underlying Density is bi-modal.

- In practice, we do not have an infinite number of samples.
- The choice of window shape is important. This effectively interpolates the data.
- If the window shape fits the local structure of the density, then Parzen windows are effective.