1 / 76

Advanced Computer Vision

Advanced Computer Vision . Chapter 7 S TRUCTURE FROM M OTION. What Is Structure from Motion?. Study of visual perception. Process of finding the three-dimensional structure of an object by analyzing local motion signals over time. A method for creating 3D models from

cheryl
Download Presentation

Advanced Computer Vision

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Advanced Computer Vision Chapter 7 STRUCTUREFROM MOTION Structure from Motion

  2. What Is Structure from Motion? • Study of visual perception. • Process of finding the three-dimensional structure of an object by analyzing local motion signals over time. • A method for creating 3D models from 2D pictures of an object. Structure from Motion

  3. Example Picture 1 Picture 2 Structure from Motion

  4. Example (cont). 3D model created from the two images Structure from Motion

  5. 7.1 Triangulation • A problem of estimating a point’s 3D location when it is seen from multiple cameras is known as triangulation. Structure from Motion

  6. Triangulation (cont). • Find the 3D point p that lies closest to all of the 3D rays corresponding to the 2D matching feature locations {xj} Structure from Motion

  7. Triangulation (cont). • Find the 3D point p that lies closest to all of the 3D rays corresponding to the 2D matching feature locations {xj} observed by cameras {Pj =Kj[Rj| tj] } tj = -Rjcj cj is the jth camera center. Structure from Motion

  8. Triangulation (cont). • It is a converse of pose estimation problem. • Given projection matrices, 3D points can be computed from their measured image positions in two or more views. Structure from Motion

  9. Triangulation (cont). Structure from Motion

  10. Triangulation (cont). Structure from Motion

  11. Triangulation (cont). Structure from Motion

  12. Triangulation (cont). • x = PX {P =K[R|t] } Structure from Motion

  13. Triangulation (cont). Figure 7.7: 3D point triangulation by finding the points p that lies nearest to all of the optical rays Structure from Motion

  14. Triangulation (cont). • The rays originate at cj in a direction • The nearest point to p on this ray, which is denoted as qj, minimizes the distance. which has a minimum at Hence, (p-cj) - Structure from Motion

  15. Triangulation (cont). • The squared distance between p and qj is • The optimal value for p, which lies closest to all of the rays, can be computed as a regular least square problem by summing over all the rj2 and finding the optimal value of p, (p-cj) - Structure from Motion

  16. Triangulation (cont). Structure from Motion

  17. Triangulation (cont). Structure from Motion

  18. Triangulation (cont). • If we use homogeneous coordinates p=(X,Y,Z,W), the resulting set of equation is homogeneous and is solved as singular value decomposition (SVD). • If we set W=1, we can use regular linear least square, but the resulting system may be singular or poorly coordinated (i.e. all of the viewing rays are parallel). Structure from Motion

  19. Singular Value Decomposition (SVD). Structure from Motion

  20. Singular Value Decomposition (SVD). Rotation Rotation Structure from Motion

  21. Singular Value Decomposition (SVD). • Solution is the eigenvector corresponding to the minimum eigenvalue of AAT • AAT= UΣVT VΣTUT = U(ΣΣT)UT • It is also the eigenvector corresponding to the minimum eigenvalue of A Structure from Motion

  22. Least Square Structure from Motion

  23. Linear Least Square Problem Structure from Motion

  24. Linear Least Square Problem • Minimize F(X): • Partial differential over X0, X1: • Solve X0, X1 by combining two equations Structure from Motion

  25. 7.2Two-Frame Structure from Motion • In 3D reconstruction we have always assumed that either 3D points position or the 3D camera poses are known in advance. Structure from Motion

  26. Two-Frame Structure from Motion (cont). Figure 7.8: Epipolar geometry: The vectors t=c1 – c0, p – c0and p-c1 are co-planar and the basic epipolar constraint expressed in terms of the pixel measurement x0and x1 Structure from Motion

  27. Two-Frame Structure from Motion (cont). • Figure shows a 3D point p being viewed from two cameras whose relative position can be encoded by a rotation R and a translation t. • We do not know anything about the camera positions, without loss of generality. • We can set the first camera at the origin c0=0 and at a canonical orientation R0=I Structure from Motion

  28. Two-Frame Structure from Motion (cont). • The observed location of point p in the first image, is mapped into the second image by the transformation : the ray direction vectors. Structure from Motion

  29. Two-Frame Structure from Motion (cont). • Taking the cross product of both the sides with t in order to annihilate it on the right hand side yields • Taking the dot product of both the sides with yields Structure from Motion

  30. Two-Frame Structure from Motion (cont). • The right hand side is triple product with two identical entries • We therefore arrive at the basic epipolar constraint : essential matrix Structure from Motion

  31. Two-Frame Structure from Motion (cont). • The essential matrix E maps a point in image 0 into a line in image 1 since Structure from Motion

  32. Two-Frame Structure from Motion (cont). • All such lines must pass through the second epipolee1, which is therefore defined as the left singular vector of E with 0 singular value, or, equivalently the projection of the vector t into image 1. • The transpose of these relationships gives us the epipolar line in the first image as and e0 as the zero value right singular vector E. Structure from Motion

  33. Two-Frame Structure from Motion (cont). Structure from Motion

  34. Two-Frame Structure from Motion (cont). • Given the relationship • If we have n corresponding measurements {(xi0,xi1)}, we can form N homogeneous equations in the elements of E= {e00…..e22} Structure from Motion

  35. Two-Frame Structure from Motion (cont). Find min||AE||, E = least eigenvector of ATA. Variants E’: enforcing the rank two constraint in E → Structure from Motion

  36. Two-Frame Structure from Motion (cont). • t is eigenvector correspended to min eignvalue under no noise: • Estimate R from t: Structure from Motion

  37. Two-Frame Structure from Motion (cont). • With , we get • Under no noise ( ): → • However, you can flip both V,U signs and still get a valid SVD: Structure from Motion

  38. Two-Frame Structure from Motion (cont). • If the measurements have noise, the terms that are product of measurement have their noise amplified by the other element in the product, which lead to poor scaling. • In order to deal with this, a suggestion is that the point coordinate should be translated and scaled so that their centroid lies at the original variance is unity; i.e. Structure from Motion

  39. Two-Frame Structure from Motion (cont). such that and n= number of points. Once the essential matrix has been computed from the transformed coordinates; the original essential matrix E can be recovered as Structure from Motion

  40. Projective Reconstruction (cont). • In the unreliable case, we do not know the calibration matrices Kj, so we cannot use the normalized ray directions. • We have access to the image coordinate xj, so essential matrix becomes: • fundamental matrix: Structure from Motion

  41. Projective Reconstruction (cont). • Just like essential matrix, fundamental matrix can be written as follow with rank 2: • And ( can not be recovered from F) Structure from Motion

  42. Projective Reconstruction (cont). • As equations on P.37, F can be written as: • Therefore, • : singular value matrix with the smallest value replaced by middle value • We can form pair projective matrices as follow and reconstruct scene by triangulation: Structure from Motion

  43. View Morphing • Application of basic two-frame structure from motion. • Also known as view interpolation. • Used to generate a smooth 3D animation from one view of a 3D scene to another. • To create such a transition: smoothly interpolate camera matrices, i.e., camera position, orientation, focal lengths. More effect is obtained by easing in and easing out camera parameters. • To generate in-between frames: establish full set of 3D correspondences or 3D models for each reference view. Structure from Motion

  44. View Morphing • Triangulate set of matched feature points in each image . • As the 3D points are re-projected into their intermediate views, pixels can be mapped from their original source images to their new views using affine projective mapping. • The final image then composited using linear blend of the two reference images as with usual morphing. Structure from Motion

  45. 7.3 Factorization • n 3D points are seen in m views • q =(u,v,1): 2D image point • p =(x,y,z,1): 3D scene point • Π : projection matrix • π : projection function • qij is the projection of the i -th point on image j • λij projective depth of qij Structure from Motion

  46. Projection Models Structure from Motion

  47. Projection Models Structure from Motion

  48. Orthographic Projection Structure from Motion

  49. Orthographic Projection Structure from Motion

  50. Perspective Projection Structure from Motion

More Related