Multigrid Multidimensional Scaling

1. Multigrid Multidimensional Scaling Michael M. Bronstein Department of Computer Science Technion – Israel Institute of Technology

2. Agenda • Applications of MDS • Numerical optimization algorithms • Motivation for multiresolution MDS methods • Multigrid MDS • Experimental results • Conclusions

3. Dimensionality reduction • Visualization • Pattern recognition • Feature extraction • Data analysis FINGER EXTENSION WRIST ROTATION Low-dimensional representation of articulated hand images, showing intrinsic data dimensionality Images: World Wide Web

4. EMBEDDING Isometric embedding • Given a surface sampled at points , and the • geodesic distances on ; • Find a mapping (isometric embedding) • such that

5. Mapmaking • Given: geodesic distances between cities on the Earth • Find: the “best” (most distance-preserving) planar map of the cities GLOBE (HEMISPHERE) PLANAR MAP Optimal planar representation of the upper hemisphere of the Earth

6. Pattern recognition ISOMETRIES OF A DEFORMABLE OBJECT ISOMETRY-INVARIANT REPRESENTATIONS (“CANONICAL FORMS”) Isometry-invariant representation of deformable objects using isometric embedding A. Elad, R. Kimmel, Proc. CVPR 2001

7. Expression-invariant face recognition • Facial expressions ~ isometries of the facial surface • Obtain expression-invariant representation using isometric embedding • Compare the canonical forms ISOMETRIC EMBEDDING FACIAL CONTOUR CROPPING FACE SUBSAMPLING DISTANCES COMPUTATION CANONICAL FORM Scheme of expression-invariant 3D face recognition based on isometric embedding A. Bronstein, M. Bronstein, R. Kimmel, Proc. AVBPA 2003; IJCV 2005

8. Stress • Given a set of distances ; • and a configuration of points in -dimensional Euclidean • space ; • Representation qualitycan be measured as the -distortion of the • distances (stress)

9. Multidimensional scaling • Stressin matrix form: • - matrix of geodesic distances (data); • - matrix of Euclidean coordinates (variable); • Multidimensional scaling (MDS) problem: • optimization variables • Optimum defined up to an isometry in

10. Minimization of the stress • Generic iterative optimization algorithm: • Start with an initial guess ; • At -st iteration, make a step of size in direction • such that • Repeat until a stopping condition is met, e.g.

11. Optimization algorithms • Gradient descent: , step size is constant or • found using line search • Newton: , step size is found using • line search • Truncated Newton: direction obtained by inexact solution of • step size is chosen to guarantee descent • Quasi-Newton: direction obtained by estimating using • the gradients ; step size is found using line • search

12. Difficulties • Non-convex and nonlinear optimization problem (local convergence) • Hessian structured but dense • High computation complexity of and • Exact line search is prohibitive for large

13. SMACOF algorithm • SMACOF: steepest descent with constant step size • where • and • Can be also written as a multiplicative update • Complexity: per iteration

14. Multiresolution methods: motivation • Data smoothness and locality (a point can be interpolated from its • neighbors) • Complexity: - MDS problem is easier on coarser resolution • Local minima: multiple resolutions improve global convergence

15. Towards multigrid MDS • Convex nonlinear optimization is equivalent to a nonlinear equation • Multigrid spirit: solve problems of the form • at different resolution levels • - residual transferred from finer resolution levels M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

16. Modified stress • Problem: the function is unbounded • Modified stress: • The penalty term forces the center of mass of to zero • With modified stress, is bounded for every finite M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

17. Multigrid components • Hierarchy of grids • Restriction and prolongation operators to transfer data and variables • from one resolution level to another • Hierarchy of optimization problems • Relaxation: steps of optimization algorithm M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

18. Coarsening schemes • In parameterization domain (suitable for parametric surfaces, e.g. • acquired by 3D scanner) • Triangulation-based (suitable for general triangulated meshes) • Farthest point sampling (based on the distances matrix; suitable for • arbitrary multidimensional data)

19. V-cycle • If (coarsest level), solve and return • Else • Relaxation • Compute • Apply MG on coarser resolution • Correction • Relaxation M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

20. Error smoothing BEFORE RELAXATION AFTER RELAXATION Error smoothing using SMACOF relaxation M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

21. Numerical experiments • Embedding of the “Swiss roll” surface – comparison of MDS algorithms • convergence in a large scale problem • Computation of canonical forms for face recognition • Sensitivity to initialization and comparison on problems of different size • Dimensionality reduction

22. Experiment I: Unrolling the Swiss roll INITIALIZATION ITERATION 1 ITERATION 3 ITERATION 4 ITERATION 2 Embedding of the Swiss roll objects into R3 using MG-MDS. N=2145 M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

23. Experiment I: Convergence comparison EXECUTION TIME (sec.) STRESS COMPLEXITY (MFLOPs) Convergence of different algorithms in the Swiss roll problem M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

24. Experiment II: Facial surface embedding INITIALIZATION ITERATION 1 ITERATION 3 ITERATION 2 Computation of a facial canonical form using MG-MDS: as few as 3 iterations are sufficient to obtain a good expression-invariant representation. N=1997 M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, NLAA, to appear

25. Experiment III: Sensitivity to initialization Performance of SMACOF and MG (V-cycle, 3 resolution levels) MDS algorithms using random initialization M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

26. Experiment III: Performance comparison Boosting obtained by multigrid MDS (V-cycle, 3 resolution levels) compared to SMACOF. Initialization by the original points M. Bronstein, A. Bronstein, R. Kimmel, I. Yavneh, Copper Mountain Conf. Multigrid Methods, 2005

27. Experiment IV: Dimensionality reduction • Two sets of random binary i.i.d. 500-dimensional vectors • Set A: • Set B: INITIALIZATION ITERATION 1 ITERATION 3 ITERATION 2 Dimensionality reduction of 500-dimensional random data: as few as 3 iterations are sufficient to obtain distinguishable clusters.

28. Conclusions • MG-MDS significantly outperforms traditional MDS algorithms • (~order of magnitude) • The improvement is more pronounced for largeN • MG-MDS appears to be less sensitive to initialization and has better • global convergence

29. References • M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "Multigrid multidimensional scaling", NLAA, to appear in 2006 • M. M. Bronstein, A. M. Bronstein, R. Kimmel, I. Yavneh, "A multigrid approach for multi- dimensional scaling", Proc. Copper Mountain Conf. Multigrid Methods, 2005. • A. M. Bronstein, M. M. Bronstein, and R. Kimmel. “Expression invariant face recognition: • faces as isometric surfaces”, in “Face Processing: Advanced Modeling and Methods”, • Academic Press, 2005. in press. • A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Three-dimensional face recognition", Intl. Journal of Computer Vision (IJCV), Vol. 64/1, pp. 5-30, August 2005. • A. M. Bronstein, M. M. Bronstein, R. Kimmel, "Expression-invariant 3D face recognition", Proc. AVBPA, Lecture Notes in Comp. Science No. 2688, Springer, pp. 62-69, 2003.