Create Presentation
Download Presentation

Download Presentation
## Linear Discriminant Analysis and Its Variations

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**Linear Discriminant Analysis and Its Variations**Abu Minhajuddin CSE 8331 Department of Statistical Science Southern Methodist University April 27, 2002**Plan…**• The Problem • Linear Discriminant Analysis • Quadratic Discriminant Analysis • Other Extensions • Evaluation of the Method • An Example • Summary**The Problem…**Training Situation: Data on p predictors, Membership of one of g groups Classification Problem: Data on p predictors Unknown group membership**The Problem…**Fisher’s Iris Data: Identify the three species?**Linear Discriminant Analysis**Classify the item xat hand to one of J groups based on measurements on p predictors. Rule: Assign xto group j that has the closest mean j = 1, 2, …, J Distance Measure: Mahalanobis Distance…. Takes the spread of the data into Consideration**Linear Discriminant Analysis**Distance Measure: For j = 1, 2, …, J, compute Assign x to the group for which dj is minimum is the pooled estimate of the covariance matrix**Linear Discriminant Analysis**Linear Discriminant Analysis …or equivalently, assign x to the group for which is a maximum. (Notice the linear form of the equation!)**Linear Discriminant Analysis**• …optimal if…. • Multivariate normal distribution for the observation in each of the groups • Equal covariance matrix for all groups • Equal prior probability for each group • Equal costs for misclassification**Linear Discriminant Analysis**Relaxing the assumption of equal prior probabilities… being the prior probability for the jth group.**Linear Discriminant Analysis**Relaxing the assumption of equal covariance matrices… result?…Quadratic Discriminant Analysis**Quadratic Discriminant Analysis**Rule: assign to group j if is the largest. Optimal if the J groups of measurements are multivariate normal**Other Extensions & Related Methods**Relaxing the assumption of normality… Kernel density based LDA and QDA Other extensions….. Regularized discriminant analysis Penalized discriminant analysis Flexible discriminant analysis**Other Extensions & Related Methods**Related Methods: Logistic regression for binary classification Multinomial logistic regression These methods models the probability of being in a class as a linear function of the predictor.**Actual group**Number of observations Predicted group A B A B nA nB n11 n21 n12 n22 Evaluations of the Methods • Classification Table (confusion matrix)**Evaluations of the Methods**Apparent Error Rate (APER): APER = # misclassified/Total # of cases ….underestimates the actual error rate. Improved estimate of APER: Holdout Method or cross validation**Actual Group**Number of Observations Predicted Group Setosa Versicolor Virginica Setosa Versicolor Virginica 50 50 50 50 0 0 0 48 1 0 2 49 An Example: Fisher’s Iris Data Table 1: Linear Discriminant Analysis (APER = 0.0200)**Actual Group**Number of Observations Predicted Group Setosa Versicolor Virginica Setosa Versicolor Virginica 50 50 50 50 0 0 0 47 1 0 3 49 An Example: Fisher’s Iris Data An Example: Fisher’s Iris Data Table 1: Quadratic Discriminant Analysis (APER = 0.0267)**Summary**• LDA is a powerful tool available for classification. • Widely implemented through various software • Theoretical properties well researched • SAS implementation available for large data sets.