Multi-view stereo

1. Multi-view stereo Many slides adapted from S. Seitz

2. What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape

3. What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape • “Images of the same object or scene” • Arbitrary number of images (from two to thousands) • Arbitrary camera positions (isolated cameras or video sequence) • Cameras can be calibrated or uncalibrated • “Representation of 3D shape” • Depth maps • Meshes • Point clouds • Patch clouds • Volumetric models • Layered models

4. Beyond two-view stereo The third view can be used for verification

5. Multiple-baseline stereo • Pick a reference image, and slide the corresponding window along the corresponding epipolar lines of all other images, using inverse depth relative to the first image as the search parameter M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993).

6. Multiple-baseline stereo Use the sum of SSD scores to rank matches

7. I1 I2 I10 Multiple-baseline stereo results M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993).

8. Summary: Multiple-baseline stereo • Pros • Using multiple images reduces the ambiguity of matching • Cons • Must choose a reference view • Occlusions become an issue for large baseline • Possible solution: use a virtual view

9. Plane Sweep Stereo • Choose a virtual view • Sweep family of planes at different depths with respect to the virtual camera input image input image composite virtual camera • each plane defines an image  composite homography R. Collins. A space-sweep approach to true multi-image matching. CVPR 1996.

10. Plane Sweep Stereo • For each depth plane • For each pixel in the composite image stack, compute the variance • For each pixel, select the depth that gives the lowest variance • Can be accelerated using graphics hardware R. Yang and M. Pollefeys. Multi-Resolution Real-Time Stereo on Commodity Graphics Hardware, CVPR 2003

11. Volumetric stereo • In plane sweep stereo, the sampling of the scene still depends on the reference view • We can use a voxel volume to get a view- independent representation

12. Volumetric stereo Scene Volume V Input Images (Calibrated) Goal: Determine occupancy, “color” of points in V

13. Discrete formulation: Voxel Coloring Discretized Scene Volume Input Images (Calibrated) • Goal: Assign RGBA values to voxels in V • photo-consistent with images

14. Complexity and computability True Scene All Scenes (CN3) Photo-Consistent Scenes Discretized Scene Volume 3 N voxels C colors

15. Voxel coloring solutions • 1. C=2 (shape from silhouettes) • Volume intersection [Baumgart 1974] • For more info: Rapid octree construction from image sequences. R. Szeliski, CVGIP: Image Understanding, 58(1):23-32, July 1993. (this paper is apparently not available online) or • W. Matusik, C. Buehler, R. Raskar, L. McMillan, and S. J. Gortler, Image-Based Visual Hulls, SIGGRAPH 2000 ( pdf 1.6 MB ) • 2. C unconstrained, viewpoint constraints • Voxel coloring algorithm [Seitz & Dyer 97] • 3. General Case • Space carving [Kutulakos & Seitz 98]

16. background + foreground background foreground - = Why use silhouettes? • Can be computed robustly • Can be computed efficiently

17. Reconstruction from silhouettes (C = 2) Binary Images How to construct 3D volume?

18. Reconstruction from silhouettes (C = 2) Binary Images • Approach: • Backproject each silhouette • Intersect backprojected volumes

19. Volume intersection • Reconstruction Contains the True Scene • But is generally not the same • In the limit (all views) get visual hull • Complement of all lines that don’t intersect S

20. Voxel algorithm for volume intersection • Color voxel black if on silhouette in every image • for M images, N3 voxels • Don’t have to search 2N3 possible scenes! O( ? ),

21. Visual Hull Results • Download data and results from http://www-cvr.ai.uiuc.edu/ponce_grp/data/visual_hull/

22. Properties of volume intersection • Pros • Easy to implement, fast • Accelerated via octrees [Szeliski 1993] or interval techniques [Matusik 2000] • Cons • No concavities • Reconstruction is not photo-consistent • Requires identification of silhouettes

23. Voxel coloring solutions • 1. C=2 (silhouettes) • Volume intersection [Baumgart 1974] • 2. C unconstrained, viewpoint constraints • Voxel coloring algorithm [Seitz & Dyer 97] • For more info: http://www.cs.washington.edu/homes/seitz/papers/ijcv99.pdf • 3. General Case • Space carving [Kutulakos & Seitz 98]

24. Voxel coloring approach 1. Choose voxel • Color if consistent • (standard deviation of pixel • colors below threshold) 2. Project and correlate Visibility Problem: in which images is each voxel visible?

25. The visibility problem Known Scene Unknown Scene • Forward Visibility • - Z-buffer • - Painter’s algorithm • known scene • Inverse Visibility?known images Which points are visible in which images?

26. Depth ordering: visit occluders first! Layers Scene Traversal Condition: depth order is the same for all input views

27. Panoramic depth ordering Cameras oriented in many different directions Planar depth ordering does not apply

28. Panoramic depth ordering Layers radiate outwards from cameras

29. Panoramic layering Layers radiate outwards from cameras

30. Panoramic layering Layers radiate outwards from cameras

31. Compatible camera configurations • Inward-Looking • cameras above scene • Outward-Looking • cameras inside scene • Depth-Order Constraint • Scene outside convex hull of camera centers

32. Calibrated image acquisition Calibrated Turntable 360° rotation (21 images) Selected Dinosaur Images Selected Flower Images

33. Voxel coloring results Dinosaur Reconstruction 72 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI Flower Reconstruction 70 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI

34. Limitations of depth ordering A view-independent depth order may not exist p q • Need more powerful general-case algorithms • Unconstrained camera positions • Unconstrained scene geometry/topology

35. Voxel coloring solutions • 1. C=2 (silhouettes) • Volume intersection [Baumgart 1974] • 2. C unconstrained, viewpoint constraints • Voxel coloring algorithm [Seitz & Dyer 97] • 3. General Case • Space carving [Kutulakos & Seitz 98] • For more info: http://www.cs.washington.edu/homes/seitz/papers/kutu-ijcv00.pdf

36. Space carving algorithm Space Carving Algorithm • Initialize to a volume V containing the true scene • Choose a voxel on the current surface • Project to visible input images • Carve if not photo-consistent • Repeat until convergence Image 1 Image N …...

37. Which shape do you get? The Photo Hull is the UNION of all photo-consistent scenes in V It is a photo-consistent scene reconstruction Tightest possible bound on the true scene V Photo Hull V True Scene

38. Space carving algorithm The Basic Algorithm is Unwieldy Complex update procedure Alternative: Multi-Pass Plane Sweep Efficient, can use texture-mapping hardware Converges quickly in practice Easy to implement Results Algorithm

39. Multi-pass plane sweep Sweep plane in each of 6 principle directions Consider cameras on only one side of plane Repeat until convergence True Scene Reconstruction

40. Multi-pass plane sweep Sweep plane in each of 6 principle directions Consider cameras on only one side of plane Repeat until convergence

41. Multi-pass plane sweep Sweep plane in each of 6 principle directions Consider cameras on only one side of plane Repeat until convergence

42. Multi-pass plane sweep Sweep plane in each of 6 principle directions Consider cameras on only one side of plane Repeat until convergence

43. Multi-pass plane sweep Sweep plane in each of 6 principle directions Consider cameras on only one side of plane Repeat until convergence

44. Multi-pass plane sweep Sweep plane in each of 6 principle directions Consider cameras on only one side of plane Repeat until convergence

45. Space carving results: african violet Input Image (1 of 45) Reconstruction Reconstruction Reconstruction

46. Space carving results: hand Input Image (1 of 100) Views of Reconstruction

47. Properties of space carving • Pros • Voxel coloring version is easy to implement, fast • Photo-consistent results • No smoothness prior • Cons • Bulging • No smoothness prior

48. Photo-consistency vs. silhouette-consistency True Scene Photo Hull Visual Hull

49. Carved visual hulls • The visual hull is a good starting point for optimizing photo-consistency • Easy to compute • Tight outer boundary of the object • Parts of the visual hull (rims) already lie on the surface and are already photo-consistent Yasutaka Furukawa and Jean Ponce, Carved Visual Hulls for Image-Based Modeling, ECCV 2006.

50. Carved visual hulls • Compute visual hull • Use dynamic programming to find rims and constrain them to be fixed • Carve the visual hull to optimize photo-consistency Yasutaka Furukawa and Jean Ponce, Carved Visual Hulls for Image-Based Modeling, ECCV 2006.