Projective structure from motion

1. Projective structure from motion Marc Pollefeys COMP 256 Some slides and illustrations from J. Ponce, …

2. Last class: Affine camera • The affine projection equations are how to find the origin? or for that matter a 3D reference point? affine projection preserves center of gravity

3. All equations can be collected for all i and j • where Orthographic factorization • The orthographic projection equations are • where Note that P and X are resp. 2mx3 and 3xn matrices and therefore the rank of x is at most 3

4. Orthographic factorization • Factorize m through singular value decomposition • An affine reconstruction is obtained as follows Closest rank-3 approximation yields MLE!

5. Tentative class schedule

6. PROJECTIVE STRUCTURE FROM MOTION • The Projective Structure from Motion Problem • Elements of Projective Geometry • Projective Structure and Motion from Two Images • Projective Motion from Fundamental Matrices • Projective Structure and Motion from Multiple Images Reading: Chapter 13.

7. Problem: estimate the m 3x4 matrices M and the n positions P from the mn correspondences p . i j ij The ProjectiveStructure-from-Motion Problem Given m perspective images of n fixed points P we can write j 2mn equations in 11m+3n unknowns Overconstrained problem, that can be solved using (non-linear) least squares!

8. If M and P are solutions, i j So are M’ and P’ where i j The Projective Ambiguity of Projective SFM When the intrinsic and extrinsic parameters are unknown and Q is an arbitrary non-singular 4x4 matrix. Q is aprojectivetransformation.

9. Projective Spaces: (Semi-Formal) Definition

10. A Model of P( R ) 3

11. Projective Subspaces and Projective Coordinates

12. Projective Subspaces and Projective Coordinates P Projective coordinates

13. Projective Subspaces Given a choice of coordinate frame Line: Plane:

14. Affine and Projective Spaces

15. Affine and Projective Coordinates

16. sin(+)sin(+) {A,B;C,D}= sin(++)sin Cross-Ratios Collinear points Pencil of planes Pencil of coplanar lines

17. Cross-Ratios and Projective Coordinates * Along a line equipped with the basis * In a plane equipped with the basis In 3-space equipped with the basis

18. Projective Transformations Bijective linear map: Projective transformation: ( = homography ) Projective transformations map projective subspaces onto projective subspaces and preserve projective coordinates. Projective transformations map lines onto lines and preserve cross-ratios.

19. Perspective Projections induce projective transformations between planes.

20. Geometric Scene Reconstruction Idea: use (A,O”,O’,B,C) as a projective basis.

21. Reprinted from “Relative Stereo and Motion Reconstruction,” by J. Ponce, T.A. Cass and D.H. Marimont, Tech. Report UIUC-BI-AI-RCV-93-07, Beckman Institute, Univ. of Illinois (1993).

22. Motion estimation from fundamental matrices Q Once M and M’ are known, P can be computed with LLS. Facts: b’ can be found using LLS.

23. Projective Structure from Motion and Factorization Factorization?? • Algorithm (Sturm and Triggs, 1996) • Guess the depths; • Factorize D; • Iterate. Does it converge? (Mahamud and Hebert, 2000)

24. Bundle adjustment - refining structure and motion • Minimize reprojection error • Maximum Likelyhood Estimation (if error zero-mean Gaussian noise) • Huge problem but can be solved efficiently (exploit sparseness)

25. Bundle adjustment • Developed in photogrammetry in 50´s

26. Non-linear least squares • Linear approximation of residual • allows quadratic approximation of sum- of-squares Minimization corresponds to finding zeros of derivative N Levenberg-Marquardt: extra term to deal with singular N (decrease/increase l if success/failure to descent) (extra term = descent term)

27. P1 P2 P3 M U1 U2 W U3 WT V 3xn (in general much larger) 12xm Bundle adjustment • Jacobian of has sparse block structure • cameras independent of other cameras, • points independent of other points im.pts. view 1 Needed for non-linear minimization

28. U-WV-1WT WT V 3xn 11xm Bundle adjustment Eliminate dependence of camera/motion parameters on structure parameters Note in general 3n >> 11m Allows much more efficient computations e.g. 100 views,10000 points, solve 1000x1000, not 30000x30000 Often still band diagonal use sparse linear algebra algorithms

29. Sequential SfM • Initialize motion from two images • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

30. (reference plane;arbitrary) Initial projective camera motion • Choose P and P´compatible with F Reconstruction up to projective ambiguity • Same for more views? • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion different projective basis

31. Initializing projective structure • Reconstruct matches in projective frame by minimizing the reprojection error • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion Non-iterative optimal solution

32. Projective pose estimation • Infere 2D-3D matches from 2D-2D matches • Compute pose from (RANSAC,6pts) X F x • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion Inliers:

33. Refining and extending structure • Refining structure • Extending structure 2-view triangulation (Iterative linear) • Initialize motion • Initialize structure • For each additional view • Determine pose of camera • Refine and extend structure • Refine structure and motion

34. Refining structure and motion • use bundle adjustment Also model radial distortion to avoid bias!

35. Hierarchical structure and motion recovery • Compute 2-view • Compute 3-view • Stitch 3-view reconstructions • Merge and refine reconstruction F T H PM

36. Metric structure and motion use self-calibration (see next class) Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views) (solution possible based on model selection)

37. Dealing with dominant planes

38. PPPgric HHgric

39. Farmhouse 3D models (note: reconstruction much larger than camera field-of-view)

40. Application: video augmentation

41. Next class: Camera calibration (and self-calibration) Reading: Chapter 2 and 3