Image and Geometry

1. Digital Signal Processing Seminar (2002) Image and Geometry Yi Ma Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC http://black.csl.uiuc.edu/~yima

2. IMAGES AND GEOMETRY:From 3D to 2D and then back to 3D GEOMETRY FOR MULTIPLE IMAGES:Without knowledge of scene GEOMETRY FOR SINGLE IMAGES:With knowledge of scene GEOMETRY FOR MULTIPLE IMAGES:With knowledge of scene APPLICATIONS: Vision, graphics, robotics, and cognition, etc. CONCLUSIONS:Open problems and future work

3. IMAGES AND GEOMETRY – A Little History of Perspective Imaging • Pinhole (perspective) imaging, in most ancient civilizations. • Euclid, perspective projection, 4th century B.C., Alexandria • Pompeii frescos, 1st century A.D. Image courtesy of C. Taylor

4. IMAGES AND GEOMETRY – A Little History of Perspective Imaging • Fillippo Brunelleschi, first Renaissance artist painted with • correct perspective,1413 • “Della Pictura”, Leone Battista Alberti, 1435 • Leonardo Da Vinci, stereopsis, shading, color, 1500s • “The scholar of Athens”, Raphael, 1518 Image courtesy of C. Taylor

5. IMAGES AND GEOMETRY – The Fundamental Problem Input: Corresponding “features” in multiple images. Output: Camera motion, camera calibration, object structure. Jana’s apartment Ignoring other pictorial cues: texture, shading, contour, etc. Image courtesy of Jana Kosecka

6. IMAGES AND GEOMETRY – History of “Modern” Geometric Vision • Chasles, formulated the two-view seven-point problem,1855 • Hesse, solved the above problem, 1863 • Kruppa, solved the two-view five-point problem, 1913 • Longuet-Higgins, the two-view eight-point algorithm, 1981 • Liu and Huang, the three-view trilinear constraints, 1986 • Huang and Faugeras, SVD based eight-point algorithm, 1989 • Triggs, the four-view quadrilinear constraints, 1995 • Ma et. al., the multiple-view rank condition, 2000

7. IMAGES AND GEOMETRY – An Uncanny deja vu? “The rise of projective geometry made such an overwhelming impression on the geometers of the first half of the nineteenth century that they tried to fit all geometric considerations into the projective scheme. ... The dictatorial regime of the projective idea in geometry was first successfully broken by the German astronomer and geometer Mobius, but the classical document of the democratic platform in geometry establishing the group of transformations as the ruling principle in any kind of geometry and yielding equal rights to independent consideration to each and any such group, is F. Klein's Erlangen program.” --- Herman Weyl, Classic Groups, 1952 Synonyms: Group = Symmetry

8. IMAGES AND GEOMETRY – Motivating Examples (Berkeley Campus) Image courtesy of Paul Debevec

9. IMAGES AND GEOMETRY – Motivating Examples (CSL Building, UIUC) Image courtesy of Kun Huang

10. IMAGES AND GEOMETRY – Are Multiple Views Necessary?

11. IMAGES AND GEOMETRY – Are Multiple Views Necessary?

12. GEOMETRY FOR MULITPLE IMAGES – A Little Notation The “hat” of a vector:

13. GEOMETRY FOR MULITPLE IMAGES – Image of a Point Homogeneous coordinates of a 3-D point Homogeneous coordinates of its 2-D image Projection of a 3-D point to an image plane

14. GEOMETRY FOR MULITPLE IMAGES – Image of a Line Homogeneous representation of a 3-D line Homogeneous representation of its 2-Dco-image Projection of a 3-D line to an image plane

15. GEOMETRY FOR MULITPLE IMAGES – Incidence Relations “Pre-images” are all incident at the corresponding features. . . .

16. M encodes exactly the 3-D information missing in one image. GEOMETRY FOR MULITPLE IMAGES – Point vs. Line Point Features Line Features

17. GEOMETRY FOR MULITPLE IMAGES – Point vs. Line

18. GEOMETRY FOR MULITPLE IMAGES – The Multiple-view Matrix Theorem [Universal Rank Condition] for images of a point on a line:

19. . . . GEOMETRY FOR MULITPLE IMAGES – Images of a Family of Lines each is an image of a (different) line in 3-D: Rank =2 Rank =1 Rank =3 . . . . . .

20. GEOMETRY FOR MULITPLE IMAGES – Mixed Features

21. GEOMETRY FOR MULITPLE IMAGES – Coplanar Features Homogeneous representation of a 3-D plane Corollary [Coplanar Features] Rank conditions on the new extended multiple view matrix remain exactly the same!

22. GEOMETRY FOR MULITPLE IMAGES – Pairwise Homography Given that a point and line features lie on a plane in 3-D space: In addition to previous constraints, it simultaneously gives homography:

23. For the jth point SVD Iteration For the ith image SVD GEOMETRY FOR MULITPLE IMAGES – Reconstruction Algorithms Given m images of n(>7) points

24. 90.840 89.820 GEOMETRY FOR MULITPLE IMAGES – Reconstruction Algorithms

25. GEOMETRY FOR MULITPLE IMAGES – Reconstruction Algorithms

26. GEOMETRY FOR SINGLE IMAGES – With Scene Knowledge • Why does an image of a symmetric object give away its structure? • Why does an image of a symmetric object give away its pose? • What else can we get from an image of a symmetric object?

27. GEOMETRY FOR SINGLE IMAGES – Symmetry related scene knowledge Symmetry captures almost all “regularities”.

28. GEOMETRY FOR SINGLE IMAGES – Hidden Images from Rotation

29. GEOMETRY FOR SINGLE IMAGES – Hidden Images from Reflection

30. GEOMETRY FOR SINGLE IMAGES – Hidden Images from Translation

31. GEOMETRY FOR SINGLE IMAGES – Symmetric Structure Definition. A set of 3-D features S is called a symmetric structure if there exists a non-trivial subgroup G of E(3) that acts on it such that for every g in G, the map is an (isometric) automorphism of S. We say the structure S has a group symmetry G.

32. GEOMETRY FOR SINGLE IMAGES – Hidden Multiple Views

33. Solving g0 from Lyapunov equations: with g’i and gi known. GEOMETRY FOR SINGLE IMAGES – Symmetric Rank Condition

34. THREE TYPES OF SYMMETRY – Pose from a Reflective Symmetry Pr

35. THREE TYPES OF SYMMETRY – Reflective Symmetry (Experiment)

36. THREE TYPES OF SYMMETRY – Reflective Symmetry (Experiment)

37. THREE TYPES OF SYMMETRY – Pose from a Rotational Symmetry

38. THREE TYPES OF SYMMETRY – Rotational Symmetry (Experiment)

39. THREE TYPES OF SYMMETRY – Rotational Symmetry (Experiment)

40. THREE TYPES OF SYMMETRY – Pose from a Translatory Symmetry

41. THREE TYPES OF SYMMETRY – Translatory Symmetry (Experiment)

42. THREE TYPES OF SYMMETRY – Translatory Symmetry (Experiment)

43. THREE TYPES OF SYMMETRY – Ambiguity in Pose Recovery “(a+b)-parameter” means there are an a-parameter family of ambiguity in R0 of g0 and a b-parameter family of ambiguity in T0 of g0. P Pr Pr N

44. GEOMETRY FOR MULTIPLE IMAGES – With Scene Knowledge • 3-D reconstruction from multiple views with symmetry is • simple, accurate and robust!

45. Symmetry on object (1) 2 (2) 1 (3) 4 (4) 3 Virtual camera-camera GEOMETRY FOR MULTIPLE IMAGES – Hidden Images in Each View

46. 2 pairs of symmetric points 2(1) 1(2) Reflective homography 4(3) 3(4) Decompose H to obtain (R’, T’, N) and T0 Solve Lyapunov equation to obtain R0. GEOMETRY FOR MULTIPLE IMAGES – Reflective Homography

47. ? GEOMETRY FOR MULTIPLE IMAGES – Alignment of Different Objects

48. GEOMETRY FOR MULTIPLE IMAGES – Scale Correction For a point p on the intersection line

49. GEOMETRY FOR MULTIPLE IMAGES – Alignment of Different Views

50. For any image x1 in the first view, its corresponding image in the second view is: GEOMETRY FOR MULTIPLE IMAGES – Scale Correction