Chapter 9

1. Chapter 9 The k-Means Algorithm and Genetic Algorithm

2. Contents • k-Means algorithm • Genetic algorithm • Rough set approach • Fuzzy set approaches Chapter 8

3. The K-Means Algorithm The K-Means algorithm is a simple yet effective statistical clustering technique. Here is the algorithm: 1. Choose a value for K, the total number of clusters to be determined. 2. Choose K instances (data points) within the dataset at random. These are the initial cluster centers. 3. Use simple Euclidean distance to assign the remaining instances to their closest cluster center. Chapter 8

4. The K-Means Algorithm 4. Use the instances in each cluster to calculate a new mean for each cluster. 5. If the new mean values are identical to the mean values of the previous iteration the process terminates. Otherwise, use the new means as cluster centers and repeat steps 3-5. Chapter 8

5. The K-Means Algorithm An Example Using K-Means Chapter 8

6. The K-Means Algorithm An Example Using K-Means Chapter 8

7. The K-Means Algorithm General Considerations Chapter 8

8. The K-Means Algorithm General Considerations Chapter 8

9. The k-Nearest Neighbor Algorithm • All instances correspond to points in the n-D space. • The nearest neighbor are defined in terms of Euclidean distance. • The target function could be discrete- or real- valued. . _ _ _ . _ . + . + . _ + xq . _ + Chapter 8

10. The k-Nearest Neighbor Algorithm • For discrete-valued, the k-NN returns the most common value among the k training examples nearest toxq. • Vonoroi diagram: the decision surface induced by 1-NN for a typical set of training examples. . _ _ _ . _ . + . + . _ + xq . _ + Chapter 8

11. Discussion on the k-NN Algorithm • The k-NN algorithm for continuous-valued target functions • Calculate the mean values of the k nearest neighbors • Distance-weighted nearest neighbor algorithm • Weight the contribution of each of the k neighbors according to their distance to the query point xq • giving greater weight to closer neighbors • Similarly, for real-valued target functions Chapter 8

12. Genetic Learning Here we present a basic genetic learning algorithm. 1. Initialize a population P of n elements, often referred to as chromosomes, as a potential solution. 2. Until a specified termination condition is satisfied: a. Use a fitness function to evaluate each element of the current solution. If an element passes the fitness criteria, it remains in P. b. The population now contains m elements (m<=n). Use genetic operators to create (n-m) new elements. Add the new elements to the population. Chapter 8

13. Genetic Learning Genetic Algorithms and Supervised Learning Chapter 8

14. Genetic Learning Genetic Algorithms and Supervised Learning Chapter 8

15. Genetic Learning Genetic Algorithms and Supervised Learning Chapter 8

16. Genetic Learning Genetic Algorithms and Supervised Learning Chapter 8

17. Genetic Learning Genetic Algorithms and... Supervised Learning Chapter 8

18. Genetic Learning Genetic Algorithms and ..Unsupervised Clustering Chapter 8

19. Genetic Learning Genetic Algorithms and Unsupervised Clustering Chapter 8

20. Genetic Learning General Considerations • Here is a list of considerations when using a problem-solving approach based on genetic learning: • Genetic algorithms are designed to find globally optimized solutions. However, there is no guarantee that any given solution is not the result of a local rather than a global optimization. • The fitness function determines the computational complexity of a genetic algorithm. A fitness function involving several calculations can be computationally expensive. Chapter 8

21. Genetic Learning General Considerations • Genetic algorithms explain their results to the extent that the fitness function is understandable. • Transforming the data to form suitable for a genetic algorithm can be a challenge. Chapter 8

22. Genetic Algorithms • GA: based on an analogy to biological evolution • Each rule is represented by a string of bits • An initial population is created consisting of randomly generated rules • Based on the notion of survival of the fittest, a new population is formed to consists of the fittest rules and their offsprings • The fitness of a rule is represented by its classification accuracy on a set of training examples • Offsprings are generated by crossover and mutation Chapter 8

23. Genetic Algorithms • Population-based technique for discovery of ....knowledge structures • Based on idea that evolution represents search for optimum solution set • Massively parallel Chapter 8

24. The Vocabulary of GAs • Population • Set of individuals, each represented by one or more strings of characters • Chromosome • The string representing an individual Chapter 8

25. The vocabulary of GAs, contd. • Gene The basic informational unit on a chromosome • Allele :The value of a specific gene • Locus : The ordinal place... on a chromosome where a specific gene is found Chapter 8

26. Genetic operators • Reproduction • Increase representations of strong individuals • Crossover • Explore the search space • Mutation • Recapture “lost” genes due to crossover Chapter 8

27. Genetic operators illustrated... Chapter 8

28. GAs rely on the concept of “fitness” • Ability of an individual to survive into the next generation • “Survival of the fittest” • Usually calculated in terms of an objective fitness function • Maximization • Minimization • Other functions Chapter 8

29. Genetic Programming • Based on adaptation and evolution • Structures undergoing adaptation are computer programs of varying size and shape • Computer programs are genetically “bred” over time Chapter 8

30. The Learning Classifier System • Rule-based knowledge discovery and concept learning tool • Operates by means of evaluation, credit assignment, and discovery applied to a population of “chromosomes” (rules) each with a corresponding “phenotype” (outcome) Chapter 8

31. Components of a Learning Classifier System • Performance • Provides interaction between environment and rule base • Performs matching function • Reinforcement • Rewards accurate classifiers • Punishes inaccurate classifiers • Discovery • Uses the genetic algorithm to search for plausible rules Chapter 8

32. Rough Set Approach • Rough sets are used to approximately or “roughly” define equivalent classes • A rough set for a given class C is approximated by two sets: • a lower approximation (certain to be in C) • an upper approximation (cannot be described as not belonging to C) Chapter 8

33. Fuzzy Set Approaches • Fuzzy logic uses truth values between 0.0 and 1.0 to represent the degree of membership (such as using fuzzy membership graph) • Attribute values are converted to fuzzy values • e.g., income is mapped into the discrete categories {low, medium, high} with fuzzy values calculated Chapter 8

34. Fuzzy Set Approaches • For a given new sample, more than one fuzzy value may apply • Each applicable rule contributes a vote for membership in the categories • Typically, the truth values for each predicted category are summed. Chapter 8

35. Chapter Summary • The K-Means algorithm is a statistical unsupervised clustering technique. • All input attributes to the algorithm must be numeric and the user is required to make a decision about..... how many clusters are to be discovered. • The algorithm begins by randomly choosing one data point to represent each cluster. • Each data instance is then placed in the cluster to which it is most similar. • New cluster centers are computed and the process continues until .....the cluster centers do not change. Chapter 8

36. Chapter Summary • The K-Means algorithm is easy to implement and understand. However, • the algorithm is not guaranteed to converge to a globally optimal solution, • lacks the ability to explain what has been found, • unable to tell which attributes are significant in determining the formed clusters. • Despite these limitations, the K-Means algorithm is among the most widely used clustering techniques. Chapter 8

37. Chapter Summary • Genetic algorithms apply the theory of evolution to inductive learning. • Genetic learning can be supervised ...or ...unsupervised • typically used for problems that cannot be solved with traditional techniques. • A standard genetic approach to learning applies a fitness function to a set of data elements to determine...... which elements survive from one generation to the next. Chapter 8

38. Chapter Summary • Those elements not surviving are used to create new instances to replace deleted elements. • In addition to being used for supervised learning and unsupervised clustering, genetic techniques can be employed in conjunction with other learning techniques. Chapter 8

39. Key Terms Affinity analysis. The process of determining which things are typically grouped together. Confidence. Given a rule of the form “If A then B,” confidence is defined as the conditional probability that B is true when A is known to be true. Crossover. A genetic learning operation that creates new population elementsby combining parts of two or more elements from the current population. Chapter 8

40. Key Terms Genetic algorithm. A data mining technique based on the theory of evolution. Mutation. A genetic learning operation that creates a new population element by randomly modifying a portion of an existing element. Selection. A genetic learning operation that adds copies of current population elements with high fitness scores to the next generation of the population. Chapter 8

41. Reference Data Mining: Concepts and Techniques (Chapter 7 Slide for textbook), Jiawei Han and Micheline Kamber, Intelligent Database Systems Research Lab, School of Computing Science, Simon Fraser University, Canada Chapter 8