1 / 55

Presenter: Derek Hoiem CS 598, Spring 2009 Jan 29, 2009

Recovering Surface Layout from a Single Image D. Hoiem, A.A. Efros, M. Hebert Robotics Institute, CMU. Presenter: Derek Hoiem CS 598, Spring 2009 Jan 29, 2009. Why worry about 3d scenes?. Reason 1: We may want to interact with the scene. Navigation. Manipulation.

carter
Download Presentation

Presenter: Derek Hoiem CS 598, Spring 2009 Jan 29, 2009

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Recovering Surface Layout from a Single ImageD. Hoiem, A.A. Efros, M. HebertRobotics Institute, CMU Presenter: Derek Hoiem CS 598, Spring 2009 Jan 29, 2009

  2. Why worry about 3d scenes?

  3. Reason 1: We may want to interact with the scene Navigation Manipulation

  4. Reason 2: We need context

  5. Reason 2: We need context

  6. 2D Object Detection

  7. What the 2D Detector Sees

  8. Computers need context too True Detection False Detections Missed Missed True Detections Local Detector: [Dalal-Triggs 2005]

  9. Context in Image Space [Torralba Murphy Freeman 2004] [Kumar Hebert 2005] [He Zemel Cerreira-Perpiñán 2004]

  10. We need 3d info to reason about 3d relationships Close Not Close

  11. How to represent scene space?

  12. How to represent scene space? Holistic Scene Space: “Gist” Torralba & Oliva 2002 Oliva & Torralba 2001

  13. How to represent scene space? Depth Map Saxena, Chung & Ng 2005, 2007

  14. Gibson’s Surface Layout • Gibson: “The elementary impressions of a visual world are those of surface and edge.” The Perception of the Visual World (1950) • Focus on texture gradients slide from Aude Oliva

  15. Gibson’s Surface Layout Surface Layout (Gibson cont.) slide from Aude Oliva

  16. Gibson’s Surface Layout Surface Layout (Gibson cont.) slide from Aude Oliva

  17. Marr’s 2½D Sketch Marr’s 2½-D Sketch Figs from Aude Oliva slide

  18. Surface Layout (this paper) • Goal: Label image into 7 Geometric Classes: • Support • Vertical • Planar: facing Left (), Center ( ),Right () • Non-planar: Solid (X), Porous or wiry (O) • Sky

  19. Our Main Challenge • Recovering 3D geometry from single 2D projection • Infinite number of possible solutions! …

  20. Our World is Structured Our World Abstract World Image Credit (left): F. Cunin and M.J. Sailor, UCSD

  21. Hansen & Riseman 1978 (VISIONS) Barrow & Tenenbaum 1978 (Intrinsic Images) Brooks 1979 (ACRONYM) Marr 1982 (2½ D Sketch) Most Early Work Tried to Manually Specify the Structure Guzman 1968 Ohta & Kanade 1978

  22. Learn the Structure of the World …

  23. Infer Most Likely Scene Unlikely Likely

  24. 1. Use All Available Cues Color, texture, image location Vanishing points, lines Texture gradient

  25. Use All Available Cues

  26. 2. Get Good Spatial Support 50x50 Patch 50x50 Patch

  27. Image Segmentation • Single segmentation won’t work • Solution: multiple segmentations …

  28. Labeling Segments … … For each segment: - Get P(good segment | data) P(label | good segment, data)

  29. Image Labeling Labeled Segmentations … Labeled Pixels

  30. Decision Trees + Adaboost High in Image? Gray? Yes No Yes No Smooth? Green? High in Image? Many Long Lines? … Yes Yes No Yes No Yes No No Blue? Very High Vanishing Point? Yes No Yes No Ground Vertical Sky Collins et al. 2002

  31. Surface Confidence Maps P(Support) P(Vertical) P(Sky) P(Planar Left) P(Planar Center) P(Planar Right) Test Image P(Non-Planar Solid) P(Non-Planar Porous)

  32. Experiments: Input Image

  33. Experiments: Ground Truth

  34. Experiments: Our Result

  35. Surface Estimates: Outdoor Avg. Accuracy Main Class: 88% Subclass: 62% Input Image Ground Truth Our Result

  36. Surface Estimates: Outdoor Ground Truth Our Result Input Image

  37. Surface Estimates: Outdoor Ground Truth Our Result Input Image

  38. Surface Estimates: Paintings Input Image Our Result

  39. Surface Estimates: Indoor Avg. Accuracy Main Class: 93% Subclass: 76% Input Image Ground Truth Our Result

  40. Failures: Reflections and Shadows Our Result Input Image

  41. Average Accuracy Main Class: 88% Subclasses: 61%

  42. Importance of Many Cues

  43. Importance of Many Cues

  44. Spatial Support Matters

  45. Automatic Photo Popup Fit Ground-Vertical Boundary with Line Segments Form Segments into Polylines Cut and Fold Labeled Image Final Pop-up Model [Hoiem Efros Hebert 2005]

  46. video

  47. Surfaces Not Enough – Need Occlusion Reasoning Image Surface Labels 3D Model

  48. Surfaces + Occlusions + Objects = Better 3D Models Surface Maps Depth, Boundaries Surfaces Occlusions Boundaries Support Horizon, Object Maps Horizon, Object Maps Viewpoint/Size Reasoning Objects and Viewpoint

  49. video 2

  50. Contributions • General principles • Learn the structure of the world • Use all available cues • Spatial support matters • Use redundancy to deal with unreliable processes (segmentation) • Results include entire spread of failure and success • First work to convincingly demonstrate single-view reconstruction

More Related