From Digital Photos to Textured 3D Models on Internet

1. From Digital Photos to Textured 3D Models on Internet Gang Xu Ritsumeikan University & 3D Media Co., Ltd.

2. On-site Demo3DM Modeler3DM Stitcher3DM Calibrator3DM Viewer

3. Applications • Cultural heritage modeling • Car & ship modeling • 3D Survey • Plant Modeling • Game • Virtual Reality • Web design

4. Outline of This Tutorial • How can 3D be recovered from 2D? • Mathematical Preparations: Solving Linear Equations and Non-linear Optimization • Camera Model and Projection • Epipolar Geometry Between Images • Linear Algorithm for 2-View SFM • Bundle Adjustment (non-linear optimization)

5. General Strategy • Step 1: Find a linear or closed-form solution • Step 2: Non-linear optimization • A linear or closed-form solution via many intermediate representations usually includes large errors. But this solution can serve as the initial guess in the non-linear optimization.

6. Part I Mathematical Preparation

7. Solving Linear Equations (1)

8. Solving Linear Equations (2) • When n=2 • When n>2, the inverse of A does not exist.

9. Solving Linear Equations (3) Best compromise Define Minimize A+ : pseudo-inverse matrix of A

10. Solving Linear Homogeneous Equations • When c=0, we cannot use x Define Minimize Under ||x||=1 Solution: Eigen vector associated with the smallest eigen value of

11. An Example (1) • Fitting a line to a 2D point set • Plane equation • Minimize

12. An Example (2) x0 e1 a e2

13. Non-Linear Optimization (1) • Minimize non-linear function

14. Non-Linear Optimization (2) • High-dimensional space • No closed-form solution • Multiple local minima • Iterative procedure, starting from an initial guess, assuming that f is differentiable everywhere.

15. An Example: Bundle Adjustment Bundle Adjustment: minimizing difference between observations and back-projections

16. 3 Common Used Algorithms • Gradient Method (1st-order derivative) • Newton Method (2nd-orde derivative) • Levenberg-Marquartd Method = Gradient+Newton, used in 3DM Modeler, 3DM Stitcher and 3DM Calibrator

17. Part II Camera and Projection

18. Pinhole Camera • Pinhole Camera lens center Image

19. Projection Formula y Y y (X,Y,Z) Focal length Z Principal point focus Mirror image Real image

20. Projection Matrix Homogeneous Coordinates

21. u Principal point x (u0,v0) v y Image Coordinate Systems Intrinsic parameters Principal point approximately at the image center A: Intrinsic matrix

22. 3D Coordinate Transform Z’ X’ Y’ Object coordinate system Rotation R Translation t Z X Y Camera coordinate system

23. From Object Coordinates to Pixel Coordinates Z’ X’ u Y’ v F

24. What Is Unknown? • Intrinsic Parameters: focal length f • Extrinsic Parameters: Rotation matrix R, translation vector t

25. Rotation Representations r/||r|| ||r|| • 3×3 Rotation MatrixR • Satisfying • There are only 3 degrees of freedom. • A rotation can be represented by a 3-vector r, with ||r|| representing the rotation amount, and r/||r|| representing the rotation axis. • Others: Euler Angles, Roll-Pitch-Yaw, Quarternion

26. Rodrigue’s Formula

27. Part III Epipolar Geometry, Essential Matrix, Fundamental Matrix and Homography Matrix

28. Epipolar Geometry Between Two Images Epipolar lines Epipolar lines Epipolar plane epipoles F F’

29. Epipolar Equation F,F’,x,x’ are coplanar. x x’ F F’ t

30. Essential Matrix E: Essential Matrix

31. Fundamental Matrix F: Fundamental matrix

32. Degenerate Cases Lead to Homography • Case 1: t=0 (no translation, only rotation. Panorama- 3DM Stitcher) • Case 2: planar scene (3DM Calibrator) • In both cases: the image can be transformed into another by a homography.

33. Determining E, F and H Matricesfrom Point Matches • E, F and H can only be determined up to scale. • F (and E) can be linearly determined from 8 or more point matches. • H can be linearly determined from 4 or more point matches.

34. Part VI A Linear Algorithm for 2-View SFM

35. Determining Translation from Essential Matrix • Assuming that the intrinsic matrix A is available • First compute Essential matrix • Determining t (up to a scale) from • There are two solutions t and -t

36. Determining Rotation R • We can consider the problem as determining R from 3 pairs of vectors before and after the rotation

37. Determining Rotation from n Pairs of Vectors R • Given (pi,pi’), i=1,…,n (n>=2), • Define and minimize • There is a linear solution using SVD. … …

38. F F’ R,t Determining 3D coordinates • Given R, t, x and x’, determine unknown scales s and s’ such that the two lines “meet” in the 3D space.

39. F F’ R,t In Front of Both Cameras • s>0, s’>0 in front of both cameras • s>0, s’<0 • s<0, s’>0 • s<0, s’<0

40. Part V Bundle Adjustment

41. Coordinate System • All space points and camera positions & camera poses are defined in a common coordinate system.

42. Match Matrix • Match Matrix w(i,j)=1, if i-th point appears in j-th image; otherwise w(i,j)=0 j 1 2 3 … i 0 1 1 1 1 2 . . 1 1 0

43. Cost Function

44. Parameters to Determine • The parameters to determine are • (Xi,Yi,Zi), i=1,…,n. Totally 3n • (tj,rj,fj), j=1,…,m. Totally 7m • Altogether 3n+7m

45. Initial Guess • LM method needs good initial guesses. • They can be obtained using the previous linear algorithm. • Needs to transform all data into the common coordinate system.

46. References • Olivier Faugeras: 3D Computer Vision: A Geometry Viewpoint, MIT Press, 1993 • Gang Xu and Zhengyou Zhang: Epipolar Geometry in Stereo, Motion and Object Recognition, Kluwer, 1996 • Richard Hartley and Andrew Zisserman: Multiple View Geometry, Cambridge Univ Press, 2000