Multiview stereo

1. Multiview stereo

2. Volumetric stereo Scene Volume V Input Images (Calibrated) Goal: Determine occupancy, “color” of points in V

3. Discrete formulation: Voxel coloring Discretized Scene Volume Input Images (Calibrated) • Goal: Assign RGBA values to voxels in V • photo-consistent with images

4. True Scene All Scenes (CN3) Photo-Consistent Scenes Complexity and computability Discretized Scene Volume 3 N voxels C colors

5. 1. Choose voxel • Color if consistent • (standard deviation of pixel • colors below threshold) 2. Project and correlate Voxel coloring Visibility Problem: in which images is each voxel visible?

6. Layers Depth ordering: occluders first! Scene Traversal Condition: depth order is the same for all input views

7. Panoramic Depth Ordering • Cameras oriented in many different directions • Planar depth ordering does not apply

8. Layers radiate outwards from cameras

9. Layers radiate outwards from cameras

10. Layers radiate outwards from cameras

11. Inward-Looking • cameras above scene • Outward-Looking • cameras inside scene Compatible Camera Configurations

12. Voxel Coloring Results Dinosaur Reconstruction 72 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI Flower Reconstruction 70 K voxels colored 7.6 M voxels tested 7 min. to compute on a 250MHz SGI

13. A view-independent depth order may not exist Limitations of Depth Ordering p q • Need more powerful general-case algorithms • Unconstrained camera positions • Unconstrained scene geometry/topology

14. Initialize to a volume V containing the true scene • Choose a voxel on the current surface • Project to visible input images • Carve if not photo-consistent • Repeat until convergence Space Carving Algorithm Image 1 Image N …...

15. p Convergence • Consistency Property • The resulting shape is photo-consistent • all inconsistent points are removed • Convergence Property • Carving converges to a non-empty shape • a point on the true scene is never removed

16. V Photo Hull Which shape do you get? V • The Photo Hull is the UNION of all photo-consistent scenes in V • It is a photo-consistent scene reconstruction • Tightest possible bound on the true scene True Scene

17. Space Carving Results: African Violet Input Image (1 of 45) Reconstruction Reconstruction Reconstruction

18. Space Carving Results: Hand Input Image (1 of 100) Views of Reconstruction

19. Multi-Camera Scene Reconstruction via Graph Cuts

20. Comparison with stereo • Much harder problem than stereo • In stereo, most scene elements are visible in both cameras • It is common to ignore occlusions • Here, almost no scene elements are visible in all cameras • Visibility reasoning is vital

21. Key issues • Visibility reasoning • Incorporating spatial smoothness • Computational tractability • Only certain energy functions can be minimized using graph cuts! • Handle a large class of camera configurations • Treat input images symmetrically

22. Approach • Problem formulation • Discrete labels, not voxels • Carefully constructed energy function • Minimizing the energy via graph cuts • Local minimum in a strong sense • Use the regularity construction • Experimental results • Strong preliminary results

23. Problem formulation • Discrete set of labels corresponding to different depths • For example, from a single camera • Camera pixel plus label = 3D point • Goal: find the best configuration • Labeling for each pixel in each camera • Minimize an energy function over configurations • Finding the exact minimum is NP-hard

24. q l = 2 l = 3 C1 r p C2 Depth labels Sample configuration

25. Energy function has 3 terms: smoothness, data, visibility • Neighborhood systems involve 3D points • Smoothness: spatial coherence (within camera) • Data: photoconsistency (between cameras) • Two pixels looking at the same scene point should see similar intensities • Visibility: prohibit certain configurations (between cameras) • A pixel in one camera can have its view blocked by a scene element visible from another camera

26. l = 2 l = 3 p C1 r Smoothness neighbors Depth labels Smoothness neighborhood

27. Smoothness term • Smoothness neighborhood involves pairs of 3D points from the same camera • We’ll assume it only depends on a pair of labels for neighboring pixels • Usual 4- or 8-connected system among pixels • Smoothness penalty for configuration f is • Vmust be a metric, i.e. robustified L1 (regularity)

28. q l = 2 l = 3 C1 p If this 3D point is visible in both cameras, pixels p and q should have similar intensities C2 Depth labels Photoconsistency constraint

29. q l = 2 l = 3 C1 p Photoconsistency neighbors C2 Depth labels Photoconsistency neighborhood

30. Data (photoconsistency) term • Photoconsistency neighborhood Nphoto • Arbitrary set of pairs of 3D points (same depth) • Current implementation: if the projection of on C2 is nearest to q • Our data penalty for configuration f is • Negative for technical reasons (regularity)

31. q l = 2 l = 3 C1 p is an impossible configuration C2 Depth labels Visibility constraint

32. q l = 2 l = 3 C1 p Visibility neighbors C2 Depth labels Visibility neighborhood

33. Visibility term • Visibility neighborhood Nvis is all pairs of 3D points that violate the visibility constraint • Arbitrary set of pairs of points at different depths • Needed for regularity • The pair of points come from different cameras • Current implementation: based on the photoconsistency neighborhood • A configuration containing any pair of 3D points in the visibility neighborhood has infinite cost

34. Input (non-binary) f Input (non-binary) f Red expansion move C2 C1 Energy minimization via expansion move algorithm • We must solve the binary energy minimization problem of finding the -expansion move that most reduces E • We only need to show that all the terms in E are regular!

35. Smoothness term is regular • True because V is a metric

36. Visibility term is regular • Consider a pair of pixels p,q • Input configuration has finite cost • Therefore A=0 • 3D points at the same depth are not in visibility neighborhood Nvis • Therefore D=0 • B,C can be 0 or , hence non-negative

37. Data term is regular

38. Tsukuba images Our results, 4 interactions

39. Comparison Best results [SS ’02] Our results, 10 interactions