Structure from Motion

1. Structure from Motion Course web page: vision.cis.udel.edu/~cv April 28, 2003  Lecture 27

2. Announcements • Readings from Trucco & Verri on optical flow for Wednesday • HW 4 due today • HW 5 due on Friday, May 9

3. Outline • Factorization for structure from motion • Homework 5

4. Factorization (Tomasi & Kanade, 1992) • Assume affine cameras—i.e., orthographic projection (minimal depth effects) • Bad results when camera motion is along optical axis • n¸ 4 point correspondences over m views • Results have affine ambiguity • Metric upgrade straightforward • Batch method:Must have all frames before computing

5. Multi-View Projection • n image points are projected from 3-D scene points over m views via where i = 1, ... , m and j = 1, ... , n. Here each Pi is a 3 x 4 matrix and each Xj is a homogeneous 4-vector

17. Multi-View Affine Projection • The last row of each Pi is (0, 0, 0, 1) for affine cameras, so we can “ignore” it and write orthographic projection as: where now each Xj is an inhomogeneous 3-vector, each Mi is 2 x 3, and each ti is a 2-vector

18. Factorization: The Reconstruction Problem • We want to estimate affine cameras fMi, tigand 3-D points Xj that minimize geometric error in image coordinates:

19. Example: Shape Reconstruction from Kanade et al.

20. Factorization: Simplifying the Problem • Normalization: We can eliminate the translation vectors ti by choosing the centroid of the image points in each image as the coordinate system origin • Working in “centered coordinates”, the minimization problem is now: • This works because the centroid of the 3-D points is preserved under affine transformations

21. 3 x n 2m x 3 Factorization: Matrix Formulation • Let the measurement matrix be: • Since , minimizing means solving:

22. Factorization: Solving with SVD • There will be no exact solution with noisy points, so we want the nearest W’ to W that is an exact solution • W’ is rank 3 since it’s the product of a 2m x 3 motion matrix M’ and a 3 x n structure matrix X’ • Use singular value decomposition to get rank 3 matrix W’ closest to W • Let SVD of W = UDVT • Then W’ = U2m£3D3£3Vn£3T, where U2m£3 is first 3 columns of U, D3£3 is upper-left 3 x 3 submatrix of D, and Vn£3T is first three columns of V

23. Factorization: Structure & Motion • Set stacked camera matrix as M’ = U2m£3sqrt(D3£3) and stacked 3-D structure matrix as X’ = sqrt(D3£3)Vn£3T so that W’ = M’X’

24. Factorization: Metric Upgrade • There is an affine ambiguity since an arbitrary 3 x 3 rank 3 matrix A can be inserted as: W’ = (M’A)(A-1X’) • Get rid of ambiguity by finding A that performs “metric rectification” • Affine camera provides orthonormality constraints on A: • Rows of M’ are unit vectors (mi¢mi = 1) • Rows of M’ are orthogonal (mi¢mj = 0)

25. HW 5 • Part 1: Implement particle filter, apply to simple image sequence • Part 2 (Graduate students only): Implement factorization algorithm with point correspondences obtained from open source feature tracker • I will provide metric upgrade function