1 / 57

Clustering Techniques

Clustering Techniques. Clustering Outline. Goal: Provide an overview of the clustering problem and introduce some of the basic algorithms. Clustering Problem Overview Clustering Techniques Hierarchical Algorithms Partitional Algorithms Genetic Algorithm Clustering Large Databases.

Download Presentation

Clustering Techniques

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Clustering Techniques

  2. Clustering Outline Goal:Provide an overview of the clustering problem and introduce some of the basic algorithms • Clustering Problem Overview • Clustering Techniques • Hierarchical Algorithms • Partitional Algorithms • Genetic Algorithm • Clustering Large Databases

  3. Clustering Examples • Segment customer database based on similar buying patterns. • Group houses in a town into neighborhoods based on similar features. • Identify new plant species • Identify similar Web usage patterns

  4. Clustering Example

  5. Size Based Geographic Distance Based Clustering Houses

  6. Clustering vs. Classification • No prior knowledge • Number of clusters • Meaning of clusters • Unsupervised learning

  7. Clustering Issues • Outlier handling • Dynamic data • Interpreting results • Evaluating results • Number of clusters • Data to be used • Scalability

  8. Impact of Outliers on Clustering

  9. Clustering Problem • Given a database D={t1,t2,…,tn} of tuples and an integer value k, the Clustering Problem is to define a mapping f:Dg{1,..,k} where each ti is assigned to one cluster Kj, 1<=j<=k. • A Cluster, Kj, contains precisely those tuples mapped to it. • Unlike classification problem, clusters are not known a priori.

  10. Types of Clustering • Hierarchical – Nested set of clusters created. • Partitional – One set of clusters created. • Incremental – Each element handled one at a time. • Simultaneous– All elements handled together. • Overlapping/Non-overlapping

  11. Hierarchical Partitional Categorical Large DB Agglomerative Divisive Clustering Approaches Clustering Sampling Compression

  12. Cluster Parameters

  13. Distance Between Clusters • Single Link: smallest distance between points • Complete Link: largest distance between points • Average Link:average distance between points • Centroid:distance between centroids

  14. Hierarchical Clustering • Clusters are created in levels actually creating sets of clusters at each level. • Agglomerative • Initially each item in its own cluster • Iteratively clusters are merged together • Bottom Up • Divisive • Initially all items in one cluster • Large clusters are successively divided • Top Down

  15. Hierarchical Algorithms • Single Link • MST Single Link • Complete Link • Average Link

  16. Dendrogram • Dendrogram: a tree data structure which illustrates hierarchical clustering techniques. • Each level shows clusters for that level. • Leaf – individual clusters • Root – one cluster • A cluster at level i is the union of its children clusters at level i+1.

  17. Levels of Clustering

  18. Agglomerative Example A B E C D Threshold of 1 2 3 4 5 A B C D E

  19. MST Example A B E C D

  20. Agglomerative Algorithm

  21. Single Link • View all items with links (distances) between them. • Finds maximal connected components in this graph. • Two clusters are merged if there is at least one edge which connects them. • Uses threshold distances at each level. • Could be agglomerative or divisive.

  22. MST Single Link Algorithm

  23. Single Link Clustering

  24. Partitional Clustering • Nonhierarchical • Creates clusters in one step as opposed to several steps. • Since only one set of clusters is output, the user normally has to input the desired number of clusters, k. • Usually deals with static sets.

  25. Partitional Algorithms • MST • Squared Error • K-Means • Nearest Neighbor • PAM • BEA • GA

  26. MST Algorithm

  27. Squared Error • Minimized squared error

  28. Squared Error Algorithm

  29. K-Means • Initial set of clusters randomly chosen. • Iteratively, items are moved among sets of clusters until the desired set is reached. • High degree of similarity among elements in a cluster is obtained. • Given a cluster Ki={ti1,ti2,…,tim}, the cluster mean is mi = (1/m)(ti1 + … + tim)

  30. K-Means Example • Given: {2,4,10,12,3,20,30,11,25}, k=2 • Randomly assign means: m1=3,m2=4 • K1={2,3}, K2={4,10,12,20,30,11,25}, m1=2.5,m2=16 • K1={2,3,4},K2={10,12,20,30,11,25}, m1=3,m2=18 • K1={2,3,4,10},K2={12,20,30,11,25}, m1=4.75,m2=19.6 • K1={2,3,4,10,11,12},K2={20,30,25}, m1=7,m2=25 • Stop as the clusters with these means are the same.

  31. K-Means Algorithm

  32. Nearest Neighbor • Items are iteratively merged into the existing clusters that are closest. • Incremental • Threshold, t, used to determine if items are added to existing clusters or a new cluster is created.

  33. Nearest Neighbor Algorithm

  34. PAM • Partitioning Around Medoids (PAM) (K-Medoids) • Handles outliers well. • Ordering of input does not impact results. • Does not scale well. • Each cluster represented by one item, called the medoid. • Initial set of k medoids randomly chosen.

  35. PAM

  36. PAM Cost Calculation • At each step in algorithm, medoids are changed if the overall cost is improved. • Cjih – cost change for an item tj associated with swapping medoid ti with non-medoid th.

  37. PAM Algorithm

  38. BEA • Bond Energy Algorithm • Database design (physical and logical) • Vertical fragmentation • Determine affinity (bond) between attributes based on common usage. • Algorithm outline: • Create affinity matrix • Convert to BOND matrix • Create regions of close bonding

  39. BEA Modified from [OV99]

  40. Genetic Algorithm Example • {A,B,C,D,E,F,G,H} • Randomly choose initial solution: {A,C,E} {B,F} {D,G,H} or 10101000, 01000100, 00010011 • Suppose crossover at point four and choose 1st and 3rd individuals: 10100011, 01000100, 00011000 • What should termination criteria be?

  41. GA Algorithm

  42. Clustering Large Databases • Most clustering algorithms assume a large data structure which is memory resident. • Clustering may be performed first on a sample of the database then applied to the entire database. • Algorithms • BIRCH • DBSCAN • CURE

  43. Desired Features for Large Databases • One scan (or less) of DB • Online • Suspendable, stoppable, resumable • Incremental • Work with limited main memory • Different techniques to scan (e.g. sampling) • Process each tuple once

  44. BIRCH • Balanced Iterative Reducing and Clustering using Hierarchies • Incremental, hierarchical, one scan • Save clustering information in a tree • Each entry in the tree contains information about one cluster • New nodes inserted in closest entry in tree

  45. Clustering Feature • CT Triple: (N,LS,SS) • N: Number of points in cluster • LS: Sum of points in the cluster • SS: Sum of squares of points in the cluster • CF Tree • Balanced search tree • Node has CF triple for each child • Leaf node represents cluster and has CF value for each subcluster in it. • Subcluster has maximum diameter

  46. BIRCH Algorithm

  47. Improve Clusters

  48. DBSCAN • Density Based Spatial Clustering of Applications with Noise • Outliers will not effect creation of cluster. • Input • MinPts – minimum number of points in cluster • Eps – for each point in cluster there must be another point in it less than this distance away.

  49. DBSCAN Density Concepts • Eps-neighborhood: Points within Eps distance of a point. • Core point: Eps-neighborhood dense enough (MinPts) • Directly density-reachable: A point p is directly density-reachable from a point q if the distance is small (Eps) and q is a core point. • Density-reachable: A point si density-reachable form another point if there is a path from one to the other consisting of only core points.

  50. Density Concepts

More Related