Fuzzy Pattern Recognition

1. Fuzzy Pattern Recognition

2. Feature Reduction Overview of Pattern Recognition • Pattern Recognition Procedure Unknown Class Label Classification (supervised) Speech/Image /Data Feature Extraction Known Clustering (unsupervised or self-organizing) Clusters Performance Criteria Cluster Validity

3. Overview of Pattern Recognition • Supervised Learning for Classification • The class label is known for a set of samples. • Find the decision boundary from the given samples. • For unknown data set, do classification • Unsupervised Learning for Clustering • Set of data is given, find the group or grouping boundary • Reinforcement Learning (Reward/Penalty) • Unkind teacher is given • Trial and Error Scheme

4. Overview of Pattern Recognition • Classification and Clustering Problem: Which class to assign Problem: How to partition How many clusters Class 1 Class 2 ? Clustering Classification

5. Overview of Pattern Recognition • Pattern Recognition Algorithm • Based on statistical approach • Parametric Approach • Bay’es Classifier with Gaussian Density • Nonlinear Boundary or Decision Function • Nonparametric Approach for Density Estimation • Parzen window • K-nearest method • Based on Neural Networks • Classifier • Multilayer Perceptron, ART, Neocogntion, … • Clustering • SOM(Self-Organizing Map)

6. Fuzzy Pattern Recognition • Classification • Rule-Based Classifier • Fuzzy Perceptron • Fuzzy K-NN Algorithm • Clustering • Fuzzy C-Mean • Possibilistic C-Mean • Fuzzy C-Shell Clustering • Fuzzy Rough Clustering • Cluster Validity • Validity Measures Based on Fuzzy Set Theory

7. Fuzzy Pattern Recognition

8. Fuzzy Classification • Rule-Based Classifier • Idea: Nonlinear Partition of Feature Space • How to find the rule from sample data. • Project the labeled training data, and design membership functions • Fuzzy clustering and projection to obtain membership function

9. Fuzzy Classification • Fuzzy K-Nearest Neighbor Algorithm • Crisp K-NN Algorithm Class 1 Class 2 Class 2 K = 3 Class 1

11. Fuzzy Classification • Fuzzy K-Nearest Neighbor Algorithm • Fuzzy K-NN Algorithm Class 1 Class 2

13. Fuzzy Nearest Prototype Classification • Crisp and Fuzzy Nearest Prototype Classification Prototype of Class 1 Prototype of Class 2 Decision Boundary

14. Crisp Version • Fuzzy Version

15. Fuzzy Perceptron • Crisp Single-Layer Perceptron (Two-class problem) Find the linear decision boundary of separable data Linear Decision Boundary

17. Fuzzy Perceptron • Fuzzy Perceptron

18. Fuzzy Perceptron • Fuzzy Perceptron • Advantage • Generalize the crisp algorithm • Elegant termination in non-separable case • Crisp case: Not terminate in finite time

19. Fuzzy Perceptron • Termination of FP • If misclassifications are all caused by very fuzzy data, then terminate the learning. • Note: FP can be combined with kernel-based method. (J.H. Chen & C.S. Chen, IEEE Trans. On NNs, 2002)

20. Fuzzy C-Mean • Clustering Objective • The aim of the iterative algorithm is to decrease the value of an objective function • Notations • Samples • Prototypes • L2-distance:

21. Fuzzy C-Mean • Crisp objective: • Fuzzy objective

22. Fuzzy C-Mean • Crisp C-Mean Algorithm • Initiate k seeds of prototypes p1, p2, …, pk • Grouping: Assign samples to their nearest prototypes Form non-overlapping clusters out of these samples • Centering: Centers of clusters become new prototypes • Repeat the grouping and centering steps, until convergence

23. Fuzzy C-Mean • Crisp C-Mean Algorithm • Grouping: Assigning samples to their nearest prototypes helps to decrease the objective • Centering: Also helps to decrease the above objective, because and equality holds only if

24. Fuzzy C-Mean • Membership matrix: Uc×n • Uijis the grade of membership of samplejwith respect to prototypei • Crisp membership: • Fuzzy membership:

25. Fuzzy C-Mean • Objective function of FCM • Introducing the Lagrange multiplier λ with respect to the constraint the objective function as:

26. Fuzzy C-Mean • Setting the partial derivatives to zero, From the 2nd equation, From this fact and the 1st equation,

27. Fuzzy C-Mean • Therefore, updating rule is

28. Fuzzy C-Mean • Setting the derivative of J with respect to pi to zero,

29. Fuzzy C-Mean • Update rule ofci: • To summarize:

30. Fuzzy C-Mean K-means Fuzzy c-means

31. Fuzzy C-Mean

32. Fuzzy C-Mean • Gustafson-Kessel Algorithm

33. Cluster Validity to Determine Number of Clusters

34. Extraction of Rule Base from Fuzzy Cluster

35. Possibilistic C-Mean • Problem of FCM • Equal Evidence = Ignorance

36. Possibilistic C-Mean • Objective Function of Fuzzy C-Mean • Constraint from Ruspini: Sum of membership of a datum over all classes should be 1. • Too restrictive condition for noisy data • Objective Function of PCM • Minimize intra-cluster distance • Make membership as large as possible

37. Possibilistic C-Mean • Necessary Condition • Determination of • Average cluster distance • Based on alpha-cut

38. Possibilistic C-Mean • Membership according to

39. Possibilistic C-Mean • Cluster Centers • Inner Product • Gustafson-Kessel (See previous page) • Spherical shell cluster

40. Possibilistic C-Mean • 2-Pass Algorithm: • Initialize PC Partition • DO Until (Change in PC Partition is Small) • Update Prototype • Update PC Partition using average cluster distances • Based on the resulted PC Partition • DO Until (Change in PC Partition is Small) • Update Prototype • Update PC Partition using alpha-cut distances

41. Possibilistic C-Mean • Advantage • Robust to noisy data • Possibly good to get the fuzzy rule base FCM-Based C-Shell PCM-Based C-Shell

42. Other Notion of Distance • Other Notion of Distance • Weights on features • Optimal Weights

43. Other Notion of Distance FCM with Euclidian Distance FCM with Adaptive Distance