Spectral Clustering

Spectral Clustering Eyal David Image Processing seminar May 2008

Lecture Outline • Motivation • Graph overview and construction • Demo • Spectral Clustering • Demo • Cool implementations

A Tutorial on Spectral Clustering\Arik Azran

Dataset exhibits complex cluster shapes • K-means performs very poorly in this space due bias toward dense spherical clusters. In the embedded space given by two leading eigenvectors, clusters are trivial to separate. Spectral Clustering Example – 2 Spirals Spectral Clustering - Derek Greene

Lecture Outline • Motivation • Graph overview and construction • Graph demo • Spectral Clustering • Spectral Clustering demo • Cool implementation

Matthias Hein and Ulrike von Luxburg August 2007

Demo (Live example)

Lecture Outline • Motivation • Graph overview and construction • Demo • Spectral Clustering • Demo • Cool implementations

Eigenvectors & Eigenvalues

Demo (Live example)

Spectral Clustering Algorithm Ng, Jordan, and Weiss • Motivation • Given a set of points • We would like to cluster them into k subsets Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm • Form the affinity matrix • Define if • Scaling parameter chosen by user • Define D a diagonal matrix whose (i,i) element is the sum of A’s row i Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm • Form the matrix • Find , the k largest eigenvectors of L • These form the the columns of the new matrix X • Note: have reduced dimension from nxn to nxk Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm • Form the matrix Y • Renormalize each of X’s rows to have unit length • Y • Treat each row of Y as a point in • Cluster into k clusters via K-means Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Algorithm • Final Cluster Assignment • Assign point to cluster j iff row i of Y was assigned to cluster j Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Why? • If we eventually use K-means, why not just apply K-means to the original data? • This method allows us to cluster non-convex regions Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Some Examples

Ng et al On Spectral clustering: analysis and algorithm

User’s Prerogative • Affinity matrix construction • Choice of scaling factor • Realistically, search over and pick value that gives the tightest clusters • Choice of k, the number of clusters • Choice of clustering method Slides from Spectral Clustering by Rebecca Nugent, Larissa Stanberry based on Ng et al On Spectral clustering: analysis and algorithm

Largest eigenvalues of Cisi/Medline data λ1 • Choose k=2 λ2 How to select k? • Eigengap: the difference between two consecutive eigenvalues. • Most stable clustering is generally given by the value k that maximises the expression Spectral Clustering - Derek Greene

Recap – The bottom line Matthias Hein and Ulrike von Luxburg August 2007

Spectral Clustering