Multi-view Stereo via Volumetric Graph-cuts

1. Multi-view Stereo via Volumetric Graph-cuts Philip H. S. Torr Department of ComputingOxford Brookes University George Vogiatzis Roberto CipollaCambridge Univ. Engineering Dept.

2. Multi-view Dense Stereo Calibrated images of Lambertian scene 3D model of scene

3. Volumetric Multi-view Dense Stereo • Two main approaches • Volumetric • Disparity (depth) map

4. Dense Stereo reconstruction problem: • Two main approaches • Volumetric • Disparity (depth) map Disparity-map

5. Shape representation • Disparity-maps • MRF formulation – good optimisation techniques exist (Graph-cuts, Loopy BP) • MRF smoothness is viewpoint dependent • Disparity is unique per pixel – only functions represented

6. Shape representation • Volumetric – e.g. Level-sets, Space carving etc. • Able to cope with non-functions • Levelsets: Local optimization • Space carving: no simple way to impose surface smoothness

7. Our approach • Cast volumetric methods in MRF framework • Use approximate surface containing the real scene surface • E.g. visual hull • Benefits: • General surfaces can be represented • No depth map merging required • Optimisation is tractable (MRF solvers) • Smoothness is viewpoint independent

8. Volumetric Graph cuts for segmentation • Volume is discretized • A binary MRF is defined on the voxels • Voxels are labelled as OBJECT and BACKGROUND • Labelling cost set by OBJECT / BACKGROUND intensity statistics • Compatibility cost set by intensity gradient Boykov and Jolly ICCV 2001

9. Volumetric Graph cuts for stereo Challenges: • What do the two labels represent • How to define cost of setting them • How to deal with occlusion • Interactions between distant voxels

10. (x) Volumetric Graph cuts 1. Outer surface 2. Inner surface (at constant offset) 3. Discretize middle volume 4. Assign photoconsistency cost to voxels

11. Volumetric Graph cuts Source Sink

12. Volumetric Graph cuts cut  3D Surface S Cost of a cut(x) dS Source [Boykov and Kolmogorov ICCV 2001] S S Sink

13. Volumetric Graph cuts Minimum cut  Minimal 3D Surface under photo-consistency metric Source [Boykov and Kolmogorov ICCV 2001] Sink

14. Photo-consistency • Occlusion 1. Get nearest point on outer surface 2. Use outer surface for occlusions 2. Discard occluded views

15. Photo-consistency • Occlusion Self occlusion

16. Photo-consistency • Occlusion Self occlusion

17. threshold on angle between normal and viewing direction threshold= ~60 Photo-consistency • Occlusion N

18. Photo-consistency Normalised cross correlation Use all remaining cameras pair wise Average all NCC scores • Score

19. Photo-consistency Average NCC = C Voxel score  = 1 - exp( -tan2[(C-1)/4] / 2 ) • Score 0    1  = 0.05 in all experiments

20. Example

21. Example - Visual Hull

22. Example - Slice

23. Example - Slice with graphcut

24. Example – 3D

25. Protrusion problem • ‘Balooning’ force • favouring bigger volumes that fill the visual hull L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

26. S V Protrusion problem (x) dS - dV • ‘Balooning’ force • favouring bigger volumes that fill the visual hull L.D. Cohen and I. Cohen. Finite-element methods for active contour models and balloons for 2-d and 3-d images. PAMI, 15(11):1131–1147, November 1993.

27. Protrusion problem

28. Protrusion problem

29. SOURCE wb wb wij h Graph wij = 4/3h2*(i+j)/2 [Boykov and Kolmogorov ICCV 2001] wb = h3 i j

30. Results • Model House

31. Results • Model House – Visual Hull

32. Results • Model House

33. Results • Stone carving

34. Results • Haniwa

35. Summary • Novel formulation for multiview stereo • Volumetric scene representation • Computationally tractable global optimisation using Graph-cuts. • Visual hull for occlusions and geometric constraint • Occlusions approximately modelled Questions ?