1 / 19

A Heuristic K-means C lustering A lgorithm by K ernel PCA

A Heuristic K-means C lustering A lgorithm by K ernel PCA. Mantao Xu and Pasi Fränti. UNIVERSITY OF JOENSUU DEPARTMENT OF COMPUTER SCIENCE JOENSUU, FINLAND. Problem Formulation.

badru
Download Presentation

A Heuristic K-means C lustering A lgorithm by K ernel PCA

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. A Heuristic K-means Clustering Algorithm by Kernel PCA Mantao Xu and Pasi Fränti UNIVERSITY OF JOENSUU DEPARTMENT OF COMPUTER SCIENCE JOENSUU, FINLAND

  2. Problem Formulation Given N data samplesX={x1, x2, …, xN}, construct the codebook C ={c1, c2, …, cM} such that mean-square-error is minimized. The class membership p (i) is

  3. Traditional K-Means Algorithm • Iterations of two steps: • assignment of each data vector with a class label • computation of cluster centroid by averaging all data vectors that are assigned to it • Characteristics: • Randomized initial partition or codebook • Convergence to a local minimum • Use of L2, L1 and L distance • Fast and easy implementation • Extensions: • Kernel K-means algorithm • EM algorithm • K-median algorithm

  4. Motivation Investigation on a clustering algorithm that : • performs the conventional K-Means algorithm in searching a solution close to the global optimum. • estimates the initial partition close to the optimal solution • applies a dissimilarity function based on the current partition instead of L2 distance

  5. Selecting initial partiton Based on kernel feature extraction approach and dynamic programming (DP): • Construct 1-D subspace by kernel PCA. • Find a suboptimal partition by DP in the 1-D subspace. • Output the partition of the DP as the initial solution to the K-means algorithm.

  6. Kernel PCA vs. PCA Sovle principal component analysis (PCA) in the reproducing kernel space F, thus implicitly depicts the irregular shape of data with a nonlinear hypercurve: PCA Kernel PCA

  7. Problem formulation of kernel PCA For the kernel PCA for the N data samples, X={xi}, solve the eigenvalue problem in F, which is assumed to be equivalent to: where the eigenvector V is a linear expansion by ans K is the kernel matrix with respect to data X

  8. Dynamic programming in kernel component direction The optimal convex partition Qk={(qj-1,qj]| j=1,,n} in the kernel component direction wcan be obtained by dynamic promgramming in terms of either MSE distortion on one-dimensional kernel component subspace (1) or in terms of MSE distortion on original feature space (2)

  9. ,G )=AddVariance Delta-MSE(x 4 2 x 2 y 1 G G y x x 2 1 4 1 2 y 3 x 3 Delta-MSE(x ,G )=RemovalVariance 4 1 Application of Delta-MSE Dissimilarity Move vector x fromcluster ito cluster j, the change of the MSE function [10] caused by this move is:

  10. Pseudocodes of the heuristic K-Means

  11. Four K-Means algorithms used in the experiments • K-D tree based K-Means: selects its initial cluster centroids from the k-bucket centers of a kd-tree structure that is recursively built by PCA. • PCA-based K-Means: estimate a sub-optimal initial partition by applying the dynamic programming in the PCA direction • KPCA-I: the proposed K-Means algorithm based on the dynamic programming criterion (1) • LFD-II: the proposed K-Means algorithm based on the dynamic programming criterion (2)

  12. Performance comparison 1 F-ratio validity index values for UCI data sets:

  13. Performance comparison 2 F-ratio validity index values for image data sets:

  14. F-ratio validity index values

  15. F-ratio validity index values

  16. F-ratio validity index values

  17. F-ratio validity index values

  18. Conclusions A new approach to the k-center clustering problem by incorporating the kernel PCA and dynamic programming. The proposed approachin general is superior to thetwo other algorithms: the PCA-based and the kd-tree basedK-Means. Gain in classification performance of the proposed approach increases with the number of clustersin comparison to two others.

  19. Further Work • Solving the k-center clustering problem by iteratively incorporating the kernel Fisher discriminant analysis and the dynamic programming technique • Solving the k-center clustering problem by boosting a decision function (conduct decision function f over X to obtain a scalar space f(X)).

More Related