Final Class: Range Data registration

1. Final Class: Range Data registration CISC4/689 Credits: Tel-Aviv University

2. The Problem • Align two partially-overlapping meshesgiven initial guessfor relative transform

3. Data Types • Point sets • Line segment sets (polylines) • Implicit curves : f(x,y,z) = 0 • Parametric curves : (x(u),y(u),z(u)) • Triangle sets (meshes) • Implicit surfaces : s(x,y,z) = 0 • Parametric surfaces (x(u,v),y(u,v),z(u,v)))

4. Motivation • Shape inspection • Motion estimation • Appearance analysis • Texture Mapping • Labeling (atlas registration)

5. Motivation • Range images registration

6. Range Scanners

7. Aligning 3D Data

8. Iterative Closest Point Algorithm • Also called ICP algorithm proposed in 1992. • Many variants have come into existence after the original algorithm proposed by Besl and Mackay.

9. Corresponding Point Set Alignment • Let M be a model point set. • Let S be a scene point set. We assume : • NM = NS. • Each point Si correspond to Mi .

10. Corresponding Point Set Alignment The objective function : The alignment is :

11. Aligning 3D Data • If correct correspondences are known, can find correct relative rotation/translation

12. Aligning 3D Data • How to find correspondences: User input? Feature detection? Signatures? • Alternative: assume closest points correspond

13. Aligning 3D Data • How to find correspondences: User input? Feature detection? Signatures? • Alternative: assume closest points correspond

14. Aligning 3D Data • Converges if starting position “close enough“

15. Closest Point • Given 2 points r1 and r2 , the Euclidean distance is: • Given a point r1 and set of points A , the Euclidean distance is:

16. Finding Matches • The sceneshape S is aligned to be in the best alignment with the model shape M. • The distance of each point s of the scene from the model is :

17. Finding Matches • Finding each match is performed in O(NM) worst case. • Given correspondence, Y we can calculate alignment • S is updated to be :

18. The Algorithm Init the error to ∞ Y = CP(M,S),e Calculate correspondence (rot,trans,d) Calculate alignment S`= rot(S)+trans Apply alignment d` = d Update error If error > threshold

19. Convergence Theorem • Correspondence error : • Alignment error:

20. ICP Variants • Variants on the following stages of ICPhave been proposed: • Selecting sample points (from one or both meshes) • Matching to points in the other mesh • Weighting the correspondences • Rejecting certain (outlier) point pairs • Assigning an error metric to the current transform • Minimizing the error metric w.r.t. transformation

21. ICP Variants • Selecting sample points (from one or both meshes). • Matching to points in the other mesh. • Weighting the correspondences. • Rejecting certain (outlier) point pairs. • Assigning an error metric to the current transform. • Minimizing the error metric w.r.t. transformation.

22. ICP Variants • Selecting sample points (from one or both meshes). • Matching to points in the other mesh using invariants. • Weighting the correspondences. • Rejecting certain (outlier) point pairs. • Assigning an error metric to the current transform. • Minimizing the error metric w.r.t. transformation.

23. ICP Variants • Selecting sample points (from one or both meshes). • Matching to points in the other mesh. • Weighting the correspondences. • Rejecting certain (outlier) point pairs. • Assigning an error metric to the current transform. • Minimizing the error metric w.r.t. transformation.

24. ICP Variants • Selecting sample points (from one or both meshes). • Matching to points in the other mesh. • Weighting the correspondences. • Rejecting certain (outlier) point pairs. • Assigning an error metric to the current transform. • Minimizing the error metric w.r.t. transformation.

25. q1 q2 p2 p1 Rejecting Pairs Inconsistent Pairs

26. ICP Variants • Selecting sample points (from one or both meshes). • Matching to points in the other mesh. • Weighting the correspondences. • Rejecting certain (outlier) point pairs. • Assigning an error metric to the current transform. • Minimizing the error metric w.r.t. transformation.

27. Error metric and minimization • Sum of squared distances between corresponding points . There exist closed form solutions for rigid body transformation : • SVD • Quaternions • Orthonoraml matrices • Dual quaternions.

28. Normal difference Curvature difference 3D Surface-to-surface Motion Analysis • Direct Shape-based method: • J. S. Duncan, et al. 1991 • J. Feldmar, et al. 1996 • Y. Wang, et al. 2000 • D. Meier,et al. 2002 • Nonrigid Shape-based method: • Nonrigid shape relationship between the before-motion and after-motion surfaces is described by the undergoing nonrigid motion. • C. Kambhamettu, et al. CVPR 1992 • C. Kambhamettu, et al. CVGIP:IU 1994 • C. Kambhamettu, et al. IVC 2003 • P. Laskov, et al. PAMI 2003

29. 3D Surface-to-surface Motion Analysis • Previous Nonrigid Shape-based Methods • A local coordinate system is constructed at each point of interest • Defined motion has no explicit physical meaning • Each point of interest is looking for its corresponding point independently • Motion consistency can not be guaranteed • New Approach of Nonrigid Shape-based Method • Nonrigid motion is modeled with a single spline-based motion field (GRBF) over the whole 3D surface. • Nonrigid shape relationship is still described in the local coordinate system constructed at each point of interest

30. Before motion After motion First fundamental form Unit normal Discriminant Motion divergence Modulus of dilation Background • At each point in the local coordinate system

31. Nonrigid shape relationship we used Nonrigid shape relationship for small deformation, additional assumption Background • Assume orthogonal parameterization (F=0)

32. From World to Local Coordinate • Principal local coordinate system • Motion transformation:

33. GRBF motion vector Nonrigid relationship Least-square error Problem Statement • Motion estimation: recover the GRBF motion • What we want to know: • What we already know: • What we should do:

34. Correspondence errors for five small-to-large paper bending deformations • Paper bending

35. Correspondence errors for five small-to-large smile deformations Correspondence errors for five small-to-large open-mouth deformations

36. Experiments Correspondences between frame 1 and Frame 6 is first estimated. Intermediate faces are reconstructed using linear interpolation, based on obtained correspondences between frame 1 and 6

37. 3D tongue Tongue Motion Analysis Sagittal+Axial Sagittal+Coronal

38. Tongue Motion Analysis A tagged MRI image. Tags are used for validation only.

39. Tongue Motion Analysis Correspondence errors for 11 tongue deformations

40. Evaluation of Structure and Nonrigid Motion (evaluation of both structure and motion) • Torso • Bullfight • More..

41. Face Motion Application • Facial Animation Parameters (FAPs) • Facial Definition Parameters (FDPs) Drive a face:movie Build and drive an Avatar:Demo Face Anatomy Motion: Movie1 (US) Movie2 (MRI)

42. REVIEW • www.cis.udel.edu/~chandra/courses.htm • Exam will only take 1.30 min., though you are given 2 hours.

43. Thank you! • Please complete Evaluations • Come to my office and.. • Take your mid-term2 • Show project progress • If you have to run to a class: show me quick progress, meet again. • 5/20 is deadline for the project unless there is a great reason for extension for few more days. • I prefer to give you an extra day or two rather than evaluating half finished product. • Its easy to come show me the demo for project evaluation. However, project html reports will be gladly accepted.