Locally Constraint Support Vector Clustering

1. Locally Constraint Support Vector Clustering Dragomir Yankov, Eamonn Keogh, Kin Fai Kan Computer Science & Eng. Dept. University of California, Riverside

2. Outline • On the need of improving the Support Vector Clustering (SVC) algorithm. Motivation • Problem formulation • Locally constrained SVC • An overview of SVC • Applying factor analysis for local outlier detection • Regularizing the decision function of SVC • Experimental evaluation

3. Motivation for improving SVC • SVC transforms the data in a high dimensional feature space, where a decision function is computed • The support-vectors define contours in the original space representing higher density regions • The method is theoretically sound and useful for detecting non-convex formations original data detected clusters

4. Motivation for improving SVC (cont) • Parametrizing SVC incorrectly may either disguise some objectively present clusters, or produce multiple unintuitive clusters • Correct parametrization is especially hard in the presence of noise (frequently encountered when learning from embedded manifolds) large kernel widths merge the clusters small kernel widths produce multiple unintuitive clusters

5. Problem formulation How can we make Support Vector Clustering: • Less susceptible to noise in the data • More resilient to imprecise parametrization

6. Locally constrained SVC – one class classification • Support Vector density estimation • Primal formulation • Dual formulation

7. Locally constrained SVC – labeling the closed contours • Support Vector Clustering – decision function • Labeling the individual classes Build an affinity matrix and find the connected components

8. Locally constrained SVC – detecting local outliers • Factor analysis: • Mixture of factor analyzers • We can adapt MFA to pinpoint local outliers Points like P1and P2 that deviate a lot from the FA are among the true outliers

9. Locally constrained SVC – regularizing the decision function • To compute the local deviation of each point we use their Mahalanobis distances with respect to the corresponding FA • New primal formulation (weighting the slack variables) • New dual formulation

10. Experimental evaluation – synthetic data • Gaussian with radial Gaussian distributions LSVC Good parameter values for LSVC are detected automatically. The right clusters are detected SVC SVC is harder to parametrize. The detected clusters are incorrect

11. Experimental evaluation – synthetic data • Swiss roll data with added Gaussian noise LSVC Most of the noise is identified as bounded SVs by LSVC. The correct clusters are detected SVC SVC tends to merge the two large clusters. With supervision the clusters are eventually identified

12. Experimental evaluation – face images • Frey face dataset LSVC LSVC discriminates the two objectively interesting manifolds embedding the data SVC Even with supervision we could not find parameters that separate the two major manifolds with SVC

13. Experimental evaluation – shape clustering • Arrowheads dataset LSVC Some of the classes are similar. There are multiple elements bridging their shape manifolds SVC LSVC achieves 73% accuracy vs 60% for SVC

14. Conclusion • The LSVC method combines both a global and a local view of the data • It computes a decision function that defines a global measure of density support • MFA complements this with a local view based on the individual analyzers • The algorithm improves significantly on the stability of SVC in the presence of noise • LSVC allows for easier automatic parameterization of one-class SVMs

15. All datasets and the code for LSVC can be obtained by writing to the first author: dyankov@cs.ucr.edu THANK YOU!