CSCI480/582 Lecture 32 Chap 6.1 3D Reconstruction Using Image Sequence Apr, 15, 2009

1. CSCI480/582 Lecture 32 Chap 6.1 3D Reconstruction Using Image Sequence Apr, 15, 2009

2. Outline • The 2D-3D problem • A perfect stereo vision condition • The critical issues in real applications • Camera calibration • Finding correspondence between images

3. The 2D-3D Problem • Given an multi-camera images of a static scene, reconstruct the 3D scene • Photo tourism demo • Given an image sequence from a moving camera of a static scene, reconstruct the 3D scene • Given an image sequence from a moving camera of an unconstrained scene, reconstruct the static content of the 3D scene • Given image sequences from multiple moving cameras of an unconstrained scene, reconstruct the 3D scene

4. Static Scene and Multiple Camera Views • An unknown static scene • Several viewpoints • 4 views up to several hundreds • ~20-50 on average

5. Sample Image Sequence [Lhuillier and Quan] • How to retrieve the 3D geometry of the object given these images?

6. A Perfect Stereo Vision Condition • Two perfect pin-hole cameras with known geometries • Pixel coordinates of the 3D point projected onto the image plane of two cameras • The 3D coordinate of the 3D point can be calculated by a simple Triangulation

7. Issues in Real Applications • Unknown cameras! • Unknown focal point location in 3D • Unknown norm vector of the imaging plane • Unknown focal length • Length distortions, digitization resolution, projection noise • Unknown pixel coordinate! • Which pixels are co-responding to the same 3D point? • And we need a lot of such pixel pairs to recover enough 3D point to describe the shape of a 3D object

8. Camera Calibration • Associate a pixel to a ray in space • Extrinsic parameters • camera position • orientation • Intrinsic parameters • Focal length • We need to at least know the relative camera geometry between the two images to build a virtual 3D scene 2D pixel  3D ray

9. The Epipolar Geometry • Given XL, XR must lie on the epipolar line determined by OL, OR, X, nL, and nR • The Epipolar constraints represented by the Fundamental Matrix between two cameras

10. The Fundamental Matrix • A 3x3 matrix F which relates corresponding points in stereo images • Given two homogeneous image coordinates, x and x’ • Fx is the epipolar line corresponding to point x • F is a rank-2 matrix, with a dof of 7

11. Solve Fundamental Matrix Linearly • For each point correspondence (x, x’) yields one equation x’TFx = 0 • As long as we have enough correspondences to determine all the unknowns in F • Let x = [u, v, 1]T, and x’ = [u’, v’, 1]T be a pair of corresponding points from two stereo images, the Fundamental matrix F=(Fij)1<=i,j<=3, then the epipolar constraints can be expressed as

12. How to Find the Correspondences? • Which subpixel locations from the two images are representing the same 3D points?

13. A Pair of ‘Good’ Correspondence YES • The quality of correspondence matching is determined by the stability of the reconstructed point location • It is even tricky to do it manually in some scenarios

14. A Pair of Bad Correspondence • How can we automate the correspondence matching process robustly? NO

15. Image Feature Detection • Rank 2 features: corner • Rank 1 features: edge

16. Finding Correspondences Given two sets of features • Geometry correlation • Ransac: Pick the best matching that provide the smallest reconstruction cost • Reconstruction cost can be designed based on transformation assumptions • Texture correlation • Match by evaluating the neighborhood texture features • Color statistics, distributions • Process mipmap to avoid local matching but global mismatch • Geometry and Texture correlation • Combine the geometry and texture features into a super descriptor vector • Then form correlation or mismatching cost functions based on the descriptor