A Tutorial on Spectral Clustering

1. A Tutorial on Spectral Clustering Ulrike von Luxburg Max Planck Institute for Biological Cybernetics Statistics and Computing, Dec. 2007, Vol. 17, No. 4 2011-07-22 Presented by Yongjin Kwon

2. Outline • Introduction • Spectral Clustering Algorithms • Two Explanations of Spectral Clustering • Graph Partitioning Point of View • Random Walks Point of View • Conclusion

3. Introduction • Clustering Algorithms • k-means / k-means++ • Mixture of Gaussians (MoG) • Hierarchical Clustering (Centroid-based, MST, Average Distance) • DBSCAN • ROCK • BIRCH • CURE • …

4. Introduction (Cont’d) (http://www.squobble.com/academic/publications/FFF_MIMO/node4.html) • Spectral Clustering • Simple, but powerful method of clustering • Requires less assumptions on the form of clusters • Outperforms the traditional approaches, such as k-means clustering.

5. Introduction (Cont’d) • Spectral Clustering? • Spectrum • Spectral analysis • Scientific or mathematic methods of analyzing something, such as light or waves, and finding the basis for them

6. Introduction (Cont’d) • Spectral Clustering? • Spectral analysis in linear algebra • Basic features of matrices : eigenpairs (eigenvalue, eigenvector) • Methods of using the eigenpairs to solve given problems • Spectral Clustering! • Methods of using the eigenvectors of some matrices to find a partition of the data such that points in the same group are similar

7. Spectral Clustering Algorithms • Similarity Graphs • ε-neighborhood graph • Connect all points whose pairwise distances are smaller than ε. • k-nearest neighbor graph • Connect two points if one is among the k-nearest neighbors of the other (and vice versa for mutualk-nearest neighbor graph). • Each edge is weighted by the similarity of their endpoints. • fully connected graph • Connect all points and weight all edges by similarity of their endpoints.

8. Spectral Clustering Algorithms (Cont’d) • In spectral clustering, the Gaussian similarity is used to repre-sent local neighborhood relationships. • : adjacency matrix of similarity graph • : degree matrix

9. Spectral Clustering Algorithms (Cont’d) related to random walk Assume the weights of edges are 1. • Graph Laplacian • unnormalized graph Laplacian : • normalized graph Laplacian • Example

10. Spectral Clustering Algorithms (Cont’d) Unnormalized Spectral Clustering

11. Spectral Clustering Algorithms (Cont’d) Normalized Spectral Clustering [Shi2000]

12. Spectral Clustering Algorithms (Cont’d) Normalized Spectral Clustering [Ng2002]

13. Two Explanations of Spectral Clustering • Graph partitioning point of view • Based on mincut problem in a similarity graph • Find a partition such that the edges between clusters have a very low weight and the edges within a cluster have high weight. • Random walks point of view • Based on random walks on the similarity graph • Find a partition such that random walk stays long within the same cluster and seldom jumps to other clusters.

14. Graph Partitioning Point of View • Mincut problem • Given a number k, find a partition which minimizes • In practice, the mincut problem results in a size imbalance in the partition.

15. Graph Partitioning Point of View (Cont’d) or . • Two common objective functions • Normalized mincut problem • Given a number k, find a partition which minimizes

16. Graph Partitioning Point of View (Cont’d) • Represent partitions by k indicator vectors . • Relationship between mincut problem and graph Laplacian • Mincut problem is converted into

17. Graph Partitioning Point of View (Cont’d) Relaxing the form of indicator vectors, normalized mincut problems can be expressed with graph Laplacian.

18. Graph Partitioning Point of View (Cont’d) • The optimization problems are NP-hard. • Relax the discreteness condition of vectors . • By the Rayleigh-Ritz theorem, the solutions of above problems are the matrix that contains the k smallest eigenvectors of the graph Laplacian and normalized one , repectively.

19. Graph Partitioning Point of View (Cont’d) k-means clustering! Note that the solutions of the relaxed optimization problems does NOT indicate which nodes are included in which groups! However, we hope that if the data are well-separate, the eigenvectors of graph Laplacians are close to piecewise constant (and close to indicator vectors). Thinking of the rows of the solution matrices as another representation of data points, k-mean clustering is a way of finding an appropriate group for each point.

20. Random Walks Point of View Assume the weights of edges are 1. • Tradition Probability Matrix • If the graph is connected and non-bipartite, then the random walk always possesses a unique stationary distribution

21. Random Walks Point of View (Cont’d) • Relationship between Ncut problem and random walks • For the random walk in the stationary distribution, • The problem of finding a partition such that random walk does not have many opportunities to jump between clusters is equiva-lent to Ncut problem due to: • Relationship between and

22. Conclusion • Spectral clustering has been made popular by several researches, and extended to many non-standard settings. • Spectral clustering has many advantages. • Requires less assumptions on the form of clusters • Be simple and efficient to implement • Has no issues of getting stuck in local minima • … • However, there are some issues when applying it. • Not trivial to choose a good similarity graph • Unstable under different parameter settings