Projective structure from motion

1. Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

2. Last class: Affine camera • The affine projection equations are how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity

3. All equations can be collected for all i and j • where Orthographic factorization • The orthographic projection equations are • where Note that P and X are resp. 2mx3 and 3xn matrices and therefore the rank of x is at most 3

4. Orthographic factorization • Factorize m through singular value decomposition • An affine reconstruction is obtained as follows Closest rank-3 approximation yields MLE!

5. Tentative class schedule

6. Further Factorization work Factorization with uncertainty Factorization for indep. moving objects Factorization for dynamic objects Perspective factorization Factorization with outliers and missing pts. (Irani & Anandan, IJCV’02) (Costeira and Kanade ‘94) (Bregler et al. 2000, Brand 2001) (Sturm & Triggs 1996, …) (Jacobs 1997 (affine), Martinek and Pajdla 2001, Aanaes 2002 (perspective))

7. Structure from motion of multiple moving objects one object: multiple objects:

8. SfM of multiple moving objects

9. Structure from motion of deforming objects (Bregler et al ’00; Brand ‘01) Extend factorization approaches to deal with dynamic shapes

10. Representing dynamic shapes (fig. M.Brand) represent dynamic shape as varying linear combination of basis shapes

11. Projecting dynamic shapes (figs. M.Brand) Rewrite:

12. Dynamic image sequences One image: (figs. M.Brand) Multiple images

13. Dynamic SfM factorization? Problem: find J so that M has proper structure

14. Dynamic SfM factorization (Bregler et al ’00) Assumption: SVD preserves order and orientation of basis shape components

15. Results (Bregler et al ’00)

16. Dynamic SfM factorization (Brand ’01) constraints to be satisfied for M constraints to be satisfied for M, use to compute J hard! (different methods are possible, not so simple and also not optimal)

17. Non-rigid 3D subspace flow (Brand ’01) • Same is also possible using optical flow in stead of features, also takes uncertainty into account

18. Results (Brand ’01)

19. Results (Brand ’01)

20. Results (Bregler et al ’01)

21. PROJECTIVE STRUCTURE FROM MOTION • The Projective Structure from Motion Problem • Elements of Projective Geometry • Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices • Projective Structure and Motion from Multiple Images Reading: Chapter 13.

22. Problem: estimate the m 3x4 matrices M and the n positions P from the mn correspondences p . i j ij The ProjectiveStructure-from-Motion Problem Given m perspective images of n fixed points P we can write j 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

23. If M and P are solutions, i j So are M’ and P’ where i j The Projective Ambiguity of Projective SFM When the intrinsic and extrinsic parameters are unknown and Q is an arbitrary non-singular 4x4 matrix. Q is aprojectivetransformation.

24. Projective Spaces: (Semi-Formal) Definition

25. A Model of P( R ) 3

26. Projective Subspaces and Projective Coordinates

27. Projective Subspaces and Projective Coordinates P Projective coordinates

28. Projective Subspaces Given a choice of coordinate frame Line: Plane:

29. Affine and Projective Spaces

30. Affine and Projective Coordinates

31. sin(+)sin(+) {A,B;C,D}= sin(++)sin Cross-Ratios Collinear points Pencil of planes Pencil of coplanar lines

32. Cross-Ratios and Projective Coordinates * Along a line equipped with the basis * In a plane equipped with the basis In 3-space equipped with the basis

33. Projective Transformations Bijective linear map: Projective transformation: ( = homography ) Projective transformations map projective subspaces onto projective subspaces and preserve projective coordinates. Projective transformations map lines onto lines and preserve cross-ratios.

34. Perspective Projections induce projective transformations between planes.

35. Geometric Scene Reconstruction Idea: use (A,O”,O’,B,C) as a projective basis.

36. Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. Report UIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

37. Motion estimation from fundamental matrices Q Once M and M’ are known, P can be computed with LLS. Facts: b’ can be found using LLS.

38. Projective Structure from Motion and Factorization Factorization?? • Algorithm (Sturm and Triggs, 1996) • Guess the depths; • Factorize D; • Iterate. Does it converge? (Mahamud and Hebert, 2000)

39. Bundle adjustment - refining structure and motion • Minimize reprojection error • Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) • Huge problem but can be solved efficiently (exploit sparseness)

40. Bundle adjustment • Developed in photogrammetry in 50´s

41. Non-linear least squares • Linear approximation of residual • allows quadratic approximation of sum- of-squares Minimization corresponds to finding zeros of derivative N Levenberg-Marquardt: extra term to deal with singular N (decrease/increase l if success/failure to descent) (extra term = descent term)

42. P1 P2 P3 M U1 U2 W U3 WT V 3xn (in general much larger) 12xm Bundle adjustment • Jacobian of has sparse block structure • cameras independent of other cameras, • points independent of other points im.pts. view 1 Needed for non-linear minimization

43. U-WV-1WT WT V 3xn 11xm Bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m Allows much more efficient computations e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 Often still band diagonal use sparse linear algebra algorithms

44. Sequential SfM • Initialize motion from two images • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

45. (reference plane;arbitrary) Initial projective camera motion • Choose P and P´compatible with F Reconstruction up to projective ambiguity • Same for more views? • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion different projective basis

46. Initializing projective structure • Reconstruct matches in projective frame by minimizing the reprojection error • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion Non-iterative optimal solution

47. Projective pose estimation • Infere 2D-3D matches from 2D-2D matches • Compute pose from (RANSAC,6pts) X F x • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion Inliers:

48. Refining and extending structure • Refining structure • Extending structure 2-view triangulation (Iterative linear) • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

49. Refining structure and motion • use bundle adjustment Also model radial distortion to avoid bias!

50. Hierarchical structure and motion recovery • Compute 2-view • Compute 3-view • Stitch 3-view reconstructions • Merge and refine reconstruction F T H PM