Loading in 2 Seconds...
Loading in 2 Seconds...
Eigenvalue solvers for data clustering and image segmentation. Andrew Knyazev Images courtesy http://www.biosci.utexas.edu/mgm/people/faculty/profiles/VIresearch.jpg and http://www.gps.caltech.edu/~edwin/molecular_vibrations.htm Supported by NSF DMS 0612751 and 0728941.
Images courtesy http://www.biosci.utexas.edu/mgm/people/faculty/profiles/VIresearch.jpg and http://www.gps.caltech.edu/~edwin/molecular_vibrations.htm
Supported by NSF DMS 0612751 and 0728941
Free transversal vibration without damping of the mass-spring system is described by the ODE system
assumption leads to the eigenproblem
Component xi describes the up-down movement of mass i.
Images courtesy http://www.gps.caltech.edu/~edwin/MoleculeHTML/AuCl4_html/AuCl4_page.html
The time independent (standing wave) equation
The Kohn-Sham equation for electronic orbitals
keeps the supercomputers busy calculating. What is next?
There is no good widely accepted definition of clustering. The traditional graph-theoretical definition is combinatorial in nature and computationally infeasible. Heuristics rule! Good open source software, e.g., METIS and CLUTO.
Clustering can be performed hierarchically by agglomeration (bottom-up) and by division (top-down).
Agglomeration clustering example
Two-way clustering, co-clustering or bi-clustering are clustering methods where not only the objects are clustered but also the features of the objects, i.e., if the data is represented in a data matrix, the rows and columns are clustered simultaneously!
Image courtesy http://www.gg.caltech.edu/~zhukov/research/spectral_clustering/spectral_clustering.htm
Partitioning means determining clusters. Partitioning can be used recursively for hierarchical division.
Many partitioning methods are known. Here we cover:
Principal Components Analysis (PCA)
PCA/spectral partitioning is known to produce high quality clusters, but is considered to be expensive as solution of large-scale eigenproblems is required. Our expertise in eigensolvers comes to the rescue!
The K-means algorithm assigns each point to the cluster whose center (called centroid) is nearest. The center is the average of all the points in the cluster.
The initial centroids are placed randomly (left), then are moved iteratively (center) until convergence (right).
The “relaxed” K-means is equivalent to the spectral clustering, discussed next.
The main idea comes from mechanical vibrations and is intuitive: in the spring-mass system the masses which are tightly connected will have the tendency to move together synchronically in low-frequency free vibrations. Analysing the signs of the components corresponding to different masses of the low-frequency vibration modes of the system allows us to determine the clusters of the masses!
A 4-degree-of-freedom system has 4 modes of vibration and 4 natural frequencies: partition into 2 clusters using the second eigenvector:
Images Courtesy: Russell, Ketteriung U.
Undirected graphs with no self-loops and no more than one edge between any two different vertices
L=K describes transversal vibrations of the spring-mass system
(as well as Kirchhoff's law for electrical circuits of resistors)
The Laplacian matrix L is symmetric with the zero smallest eigenvalue and constant eigenvector (free boundary). The second eigenvector, called the Fiedler vector, describes the partitioning.
Example Courtesy: Blelloch CMU
Rows sum to zero
Image pixels serve as graph vertices. Weighted graph edges are computed by comparing pixel colours.
Here is an example displaying 4 Fiedler vectors of an image:
We generate a sparse Laplacian, by comparing neighbouring pixels here when computing the weights for the edges, getting a sparse Laplacian (with a small fill-in). Megapixel images lead to eigenproblems with millions of degrees of freedom.