1 / 37

Stereo Matching

Stereo Matching. Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski. Stereo Matching. Given two or more images of the same scene or object, compute a representation of its shape What are some possible applications?. Face modeling.

golda
Download Presentation

Stereo Matching

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Stereo Matching Vision for GraphicsCSE 590SS, Winter 2001Richard Szeliski

  2. Stereo Matching • Given two or more images of the same scene or object, compute a representation of its shape • What are some possible applications? Vision for Graphics

  3. Face modeling • From one stereo pair to a 3D head model[Frederic Deverney, INRIA] Vision for Graphics

  4. Z-keying: mix live and synthetic • Takeo Kanade, CMU (Stereo Machine) Vision for Graphics

  5. Virtualized RealityTM • Takeo Kanade, CMU • collect video from 50+ stream • reconstruct 3D model sequenceshttp://www.cs.cmu.edu/afs/cs/project/VirtualizedR/www/VirtualizedR.html Vision for Graphics

  6. Virtualized RealityTM • Takeo Kanade, CMU • generate new video • steerable version used for SuperBowl XXV“eye vision” system Vision for Graphics

  7. View Interpolation • Given two images with correspondences, morph (warp and cross-dissolve) between them [Chen & Williams, SIGGRAPH’93] • input depth image novel view[Matthies,Szeliski,Kanade’88] Vision for Graphics

  8. More view interpolation • Spline-based depth mapinput depth image novel view • [Szeliski & Kang ‘95] Vision for Graphics

  9. View Morphing • Morph between pair of images using epipolar geometry [Seitz & Dyer, SIGGRAPH’96] Vision for Graphics

  10. Additional applications • Real-time people tracking (systems from Pt. Gray Research and SRI) • “Gaze” correction for video conferencing [Ott,Lewis,Cox InterChi’93] • Other ideas? Vision for Graphics

  11. Stereo Matching • Given two or more images of the same scene or object, compute a representation of its shape • What are some possible representations? • depth maps • volumetric models • 3D surface models • planar (or offset) layers Vision for Graphics

  12. Stereo Matching • What are some possible algorithms? • match “features” and interpolate • match edges and interpolate • match all pixels with windows (coarse-fine) • use optimization: • iterative updating • dynamic programming • energy minimization (regularization, stochastic) • graph algorithms Vision for Graphics

  13. Outline (remainder of talk) • Image rectification • Matching criteria • Local algorithms (aggregation) • iterative updating • Optimization algorithms: • energy (cost) formulation • Markov Random Fields • mean-field, stochastic, and graph algorithms Vision for Graphics

  14. epipolar line epipolar plane viewing ray Stereo: epipolar geometry • Match features along epipolar lines Vision for Graphics

  15. Stereo: epipolar geometry • for two images (or images with collinear camera centers), can find epipolar lines • epipolar lines are the projection of the pencil of planes passing through the centers • Rectification: warping the input images (perspective transformation) so that epipolar lines are horizontal Vision for Graphics

  16. Rectification • Project each image onto same plane, which is parallel to the epipole • Resample lines (and shear/stretch) to place lines in correspondence, and minimize distortion • [Zhang and Loop, MSR-TR-99-21] Vision for Graphics

  17. Rectification Vision for Graphics

  18. Rectification Vision for Graphics

  19. Matching criteria • Raw pixel values (correlation) • Band-pass filtered images [Jones & Malik 92] • “Corner” like features [Zhang, …] • Edges [many people…] • Gradients [Seitz 89; Scharstein 94] • Rank statistics [Zabih & Woodfill 94] Vision for Graphics

  20. Finding correspondences • apply feature matching criterion (e.g., correlation or Lucas-Kanade) at all pixels simultaneously • search only over epipolar lines (many fewer candidate positions) Vision for Graphics

  21. Image registration (revisited) • How do we determine correspondences? • block matching or SSD (sum squared differences)d is the disparity (horizontal motion) • How big should the neighborhood be? Vision for Graphics

  22. Neighborhood size • Smaller neighborhood: more details • Larger neighborhood: fewer isolated mistakes • w = 3 w = 20 Vision for Graphics

  23. Stereo: certainty modeling • Compute certainty map from correlations • input depth map certainty map Vision for Graphics

  24. input image composite virtual camera Plane Sweep Stereo • Sweep family of planes through volume  projective re-sampling of (X,Y,Z) • each plane defines an image  composite homography Vision for Graphics

  25. Plane Sweep Stereo • For each depth plane • compute composite (mosaic) image — mean • compute error image — variance • convert to confidence and aggregate spatially • Select winning depth at each pixel Vision for Graphics

  26. Plane sweep stereo • Re-order (pixel / disparity) evaluation loopsfor every pixel, for every disparity for every disparity for every pixel compute cost compute cost Vision for Graphics

  27. Stereo matching framework • For every disparity, compute raw matching costsWhy use a robust function? • occlusions, other outliers • Can also use alternative match criteria Vision for Graphics

  28. Stereo matching framework • Aggregate costs spatially • Here, we are using a box filter(efficient moving averageimplementation) • Can also use weighted average,[non-linear] diffusion… Vision for Graphics

  29. Stereo matching framework • Choose winning disparity at each pixel • Can interpolate to sub-pixel accuracy Vision for Graphics

  30. Traditional Stereo Matching • Advantages: • gives detailed surface estimates • fast algorithms based on moving averages • sub-pixel disparity estimates and confidence • Limitations: • narrow baseline  noisy estimates • fails in textureless areas • gets confused near occlusion boundaries Vision for Graphics

  31. Stereo with Non-Linear Diffusion • Problem with traditional approach: • gets confused near discontinuities • New approach: • use iterative (non-linear) aggregation to obtain better estimate • provably equivalent to mean-field estimate of Markov Random Field Vision for Graphics

  32. Linear diffusion • Average energy with neighbors • window diffusion Vision for Graphics

  33. Linear diffusion • Average energy with neighbors + starting value • window diffusion Vision for Graphics

  34. Non-linear diffusion • Stopping criterion: • only update (x,y) column is entropy goes down (distribution is more peaked) Vision for Graphics

  35. Summary • Applications • Image rectification • Matching criteria • Local algorithms (aggregation) • area-based; iterative updating • Optimization algorithms: • energy (cost) formulation • Markov Random Fields • mean-field; dynamic programming; • stochastic; graph algorithms Vision for Graphics

  36. More stereo…(next 2 lectures) • Multi-image stereo • Volumetric techniques • Graph cuts • Transparency • Surfaces and level sets Vision for Graphics

  37. Bibliography • See the references in the readings… • D. Scharstein and R. Szeliski. Stereo matching with nonlinear diffusion. International Journal of Computer Vision, 28(2):155-174, July 1998 • R. Szeliski. Stereo algorithms and representations for image-based rendering. In British Machine Vision Conference (BMVC'99), volume 2, pages 314-328, Nottingham, England, September 1999. • R. Szeliski and R. Zabih. An experimental comparison of stereo algorithms. In International Workshop on Vision Algorithms, pages 1-19, Kerkyra, Greece, September 1999. Vision for Graphics

More Related