Linear discriminant analysis and its variations
Download
1 / 20

Linear Discriminant Analysis and Its Variations - PowerPoint PPT Presentation


  • 186 Views
  • Updated On :

Linear Discriminant Analysis and Its Variations. Abu Minhajuddin CSE 8331. Department of Statistical Science Southern Methodist University April 27, 2002. Plan…. The Problem Linear Discriminant Analysis Quadratic Discriminant Analysis Other Extensions Evaluation of the Method

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Linear Discriminant Analysis and Its Variations' - zev


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Linear discriminant analysis and its variations l.jpg
Linear Discriminant Analysis and Its Variations

Abu Minhajuddin

CSE 8331

Department of Statistical Science

Southern Methodist University

April 27, 2002


Slide2 l.jpg
Plan…

  • The Problem

  • Linear Discriminant Analysis

  • Quadratic Discriminant Analysis

  • Other Extensions

  • Evaluation of the Method

  • An Example

  • Summary


The problem l.jpg
The Problem…

Training Situation:

Data on p predictors,

Membership of one of g groups

Classification Problem:

Data on p predictors

Unknown group membership


The problem4 l.jpg
The Problem…

Fisher’s Iris Data: Identify the three species?


Linear discriminant analysis l.jpg
Linear Discriminant Analysis

Classify the item xat hand to one of J groups based on measurements on p predictors.

Rule: Assign xto group j that has the closest mean

j = 1, 2, …, J

Distance Measure: Mahalanobis Distance….

Takes the spread of the data into Consideration


Linear discriminant analysis6 l.jpg
Linear Discriminant Analysis

Distance Measure:

For j = 1, 2, …, J, compute

Assign x to the group for which dj is minimum

is the pooled estimate of the covariance matrix


Linear discriminant analysis7 l.jpg
Linear Discriminant Analysis

Linear Discriminant Analysis

…or equivalently, assign x to the group for which

is a maximum.

(Notice the linear form of the equation!)


Linear discriminant analysis8 l.jpg
Linear Discriminant Analysis

  • …optimal if….

  • Multivariate normal distribution for the observation in each of the groups

  • Equal covariance matrix for all groups

  • Equal prior probability for each group

  • Equal costs for misclassification


Linear discriminant analysis9 l.jpg
Linear Discriminant Analysis

Relaxing the assumption of equal prior probabilities…

being the prior probability for the jth group.


Linear discriminant analysis10 l.jpg
Linear Discriminant Analysis

Relaxing the assumption of equal covariance matrices…

result?…Quadratic Discriminant Analysis


Quadratic discriminant analysis l.jpg
Quadratic Discriminant Analysis

Rule: assign to group j if is the largest.

Optimal if the J groups of measurements are multivariate normal


Other extensions related methods l.jpg
Other Extensions & Related Methods

Relaxing the assumption of normality…

Kernel density based LDA and QDA

Other extensions…..

Regularized discriminant analysis

Penalized discriminant analysis

Flexible discriminant analysis


Other extensions related methods13 l.jpg
Other Extensions & Related Methods

Related Methods:

Logistic regression for binary classification

Multinomial logistic regression

These methods models the probability of being in a class as a linear function of the predictor.


Evaluations of the methods l.jpg

Actual group

Number of observations

Predicted group

A

B

A

B

nA

nB

n11

n21

n12

n22

Evaluations of the Methods

  • Classification Table (confusion matrix)


Evaluations of the methods15 l.jpg
Evaluations of the Methods

Apparent Error Rate (APER):

APER = # misclassified/Total # of cases

….underestimates the actual error rate.

Improved estimate of APER:

Holdout Method or cross validation


An example fisher s iris data l.jpg

Actual Group

Number of Observations

Predicted Group

Setosa

Versicolor

Virginica

Setosa

Versicolor

Virginica

50

50

50

50

0

0

0

48

1

0

2

49

An Example: Fisher’s Iris Data

Table 1: Linear Discriminant Analysis

(APER = 0.0200)


An example fisher s iris data17 l.jpg

Actual Group

Number of Observations

Predicted Group

Setosa

Versicolor

Virginica

Setosa

Versicolor

Virginica

50

50

50

50

0

0

0

47

1

0

3

49

An Example: Fisher’s Iris Data

An Example: Fisher’s Iris Data

Table 1: Quadratic Discriminant Analysis

(APER = 0.0267)




Summary l.jpg
Summary

  • LDA is a powerful tool available for classification.

    • Widely implemented through various software

    • Theoretical properties well researched

    • SAS implementation available for large data sets.


ad