1 / 42

Unsupervised learning

Unsupervised learning. David Kauchak CS 451 – Fall 2013. Administrative. Final project Schedule fo r the rest of the semester. Unsupervised learning. Unsupervised learning: given data, i.e. examples, but no labels. K -means. Start with some initial cluster centers Iterate:

tess
Download Presentation

Unsupervised learning

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Unsupervised learning David KauchakCS 451 – Fall 2013

  2. Administrative Final project Schedule for the rest of the semester

  3. Unsupervised learning Unsupervised learning: given data, i.e. examples, but no labels

  4. K-means Start with some initial cluster centers Iterate: • Assign/cluster each example to closest center • Recalculate centers as the mean of the points in a cluster

  5. K-means: an example

  6. K-means: Initialize centers randomly

  7. K-means: assign points to nearest center

  8. K-means: readjust centers

  9. K-means: assign points to nearest center

  10. K-means: readjust centers

  11. K-means: assign points to nearest center

  12. K-means: readjust centers

  13. K-means: assign points to nearest center No changes: Done

  14. K-means variations/parameters Initial (seed) cluster centers Convergence • A fixed number of iterations • partitions unchanged • Cluster centers don’t change K!

  15. How Many Clusters? Number of clusters Kmust be provided How should we determine the number of clusters? How did we deal with models becoming too complicated previously? too few too many

  16. Many approaches Regularization!!! Statistical test

  17. k-means loss revisited K-means is trying to minimize: What happens when k increases?

  18. k-means loss revisited K-means is trying to minimize: Loss goes down! Making the model more complicated allows us more flexibility, but can “overfit” to the data

  19. k-means loss revisited K-means is trying to minimize: (where N = number of points) 2 regularization options What effect will this have?Which will tend to produce smaller k?

  20. k-means loss revisited (where N = number of points) 2 regularization options AIC penalizes increases in K more harshly Both require a change to the K-means algorithm Tend to work reasonably well in practice if you don’t know K

  21. Statistical approach Assume data is Gaussian (i.e. spherical) Test for this • Testing in high dimensions doesn’t work well • Testing in lower dimensions does work well ideas?

  22. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian

  23. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian What will this look like projected to 1-D?

  24. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian

  25. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian What will this look like projected to 1-D?

  26. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian

  27. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian What will this look like projected to 1-D?

  28. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian Solution?

  29. Project to one dimension and check For each cluster, project down to one dimension • Use a statistical test to see if the data is Gaussian Chose the dimension of the projection as the dimension with highest variance

  30. On synthetic data Split too far

  31. Compared to other approaches http://cs.baylor.edu/~hamerly/papers/nips_03.pdf

  32. K-Means time complexity Variables: K clusters, n data points, m features/dimensions, I iterations What is the runtime complexity? • Computing distance between twopoints (e.g. euclidean) • Reassigning clusters • Computing new centers • Iterate…

  33. K-Means time complexity Variables: K clusters, n data points, m features/dimensions, I iterations What is the runtime complexity? • Computing distance between two points is O(m) where m is the dimensionality of the vectors/number of features. • Reassigning clusters: O(Kn) distance computations, or O(Knm) • Computing centroids: Each points gets added once to some centroid: O(nm) • Assume these two steps are each done once for I iterations: O(Iknm) In practice, K-means converges quickly and is fairly fast

  34. What Is A Good Clustering? Internal criterion: A good clustering will produce high quality clusters in which: • the intra-class (that is, intra-cluster) similarity is high • the inter-class similarity is low How would you evaluate clustering?

  35. Common approach: use labeled data Use data with known classes • For example, document classification data data label If we clustered this data (ignoring labels) what would we like to see? Reproduces class partitions How can we quantify this?

  36. Common approach: use labeled data Purity, the proportion of the dominant class in the cluster         Cluster I Cluster II Cluster III Cluster I: Purity = 1/4 (max(3, 1, 0)) = 3/4 Cluster II: Purity = 1/6 (max(1, 4, 1)) = 4/6 Overall purity? Cluster III: Purity = 1/5 (max(2, 0, 3)) = 3/5

  37. Overall purity Cluster average: Weighted average: Cluster I: Purity = 1/4 (max(3, 1, 0)) = 3/4 Cluster II: Purity = 1/6 (max(1, 4, 1)) = 4/6 Cluster III: Purity = 1/5 (max(2, 0, 3)) = 3/5

  38. Purity issues… Purity, the proportion of the dominant class in the cluster Good for comparing two algorithms, but not understanding how well a single algorithm is doing, why? • Increasing the number of clusters increases purity

  39. Purity isn’t perfect Which is better based on purity? Which do you think is better? Ideas?

  40. Common approach: use labeled data Average entropy of classes in clusters where p(classi) is proportion of class i in cluster

  41. Common approach: use labeled data Average entropy of classes in clusters entropy?

  42. Common approach: use labeled data Average entropy of classes in clusters

More Related