- 102 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Non-Parametric Learning' - reese-roberson

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Parametric versus Non-Parametric

- Previous lectures on MLE learning assumed a functional form for the probability distribution.
- We now consider an alternative non-parametric method based on window function.

Non-Parametric

- It is hard to develop probability models for some data.
- Example: estimate the distribution of annual rainfall in the U.S.A. Want to model p(x,y) – probability that a raindrop hits a position (x,y).
- Problems: (i) multi-modal density is difficult for parametric models, (ii) difficult/impossible to collect enough data at each point (x,y).

Intuition

- Assume that the probability density is locally smooth.
- Goal: estimate the class density model p(x) from data
- Method 1: Windows based on points x in space.

Windows

- For each point x, form a window centred at x with volume Count the number of samples that fall in the window.
- Probability density is estimated as:

Non-Parametric

- Goal: to design a sequence of windows

so that at each point x

- (f(x) is the true density).
- Conditions for window design:
- increasing spatial resolution.

(ii) many samples at each point

(iii)

Two Design Methods

- Parzen Window: Fix window size:
- K-NN: Fix no. samples in window:

Parzen Window

- Parzen window uses a window function
- Example:
- (i) Unit hypercube:

and 0 otherwise.

- (ii) Gaussian in d-dimensions.

Parzen Windows

- No. of samples in the hypercube is
- Volume
- The estimate of the distribution is:
- More generally, the window interpolates the data.

Parzen Window Example

- Estimate a density with five modes using Gaussian windows at scales h=1,0.5, 0.2.

Convergence Proof.

- We will show that the Parzen window estimator converges to the true density at each point x with increasing number of samples.

Proof Strategy.

- Parzen distribution

is a random variable which depends on the samples used to estimate it.

- We have to take the expectation of the distribution with respect to the samples.
- We show that the expected value of the Parzen distribution will be the true distribution. And the expected variance of the Parzen distribution will tend to 0 as no. samples gets large.

Convergence of the Mean

- Result follows.

Convergence of Variance

- Variance:

Example of Parzen Window

- Underlying density is Gaussian. Window volume decreases as

Example of Parzen Window

- Underlying Density is bi-modal.

Parzen Window and Interpolation.

- In practice, we do not have an infinite number of samples.
- The choice of window shape is important. This effectively interpolates the data.
- If the window shape fits the local structure of the density, then Parzen windows are effective.

Download Presentation

Connecting to Server..