3D-Based Reasoning with Blocks, Support, and Stability
This presentation is the property of its rightful owner.
Sponsored Links
1 / 33

3D-Based Reasoning with Blocks, Support, and Stability PowerPoint PPT Presentation


  • 41 Views
  • Uploaded on
  • Presentation posted in: General

3D-Based Reasoning with Blocks, Support, and Stability. Zhaoyin Jia. School of Electrical and Computer Engineering Cornell University. Computer Vision with RGB-D. Pose Recognition J. Shotton et al. 2011; G. Girshick et al. 2013. Activity Detection

Download Presentation

3D-Based Reasoning with Blocks, Support, and Stability

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


3d based reasoning with blocks support and stability

3D-Based Reasoning with Blocks, Support, and Stability

Zhaoyin Jia

School of Electrical and Computer Engineering

Cornell University


Computer vision with rgb d

Computer Vision with RGB-D

Pose Recognition

J. Shotton et al. 2011; G. Girshick et al. 2013.

Activity Detection

J. Sung et al. 2012; H. Koppula et al. 2013.

Object Recognition

K. Lai et al. 2011; A. Janoch et al. 2011

3D Scene Labeling

H. Koppula, et al. 2011; N. Silberman et al 2011, 2012.

Jia, Gallagher, Saxena and Chen


Rgb d images

RGB-D Images

Jia, Gallagher, Saxena and Chen


3d reasoning on rgb d images

3D Reasoning on RGB-D Images

  • Free Space:

    • objects can be placed in empty spaces.

  • Physical Stability:

    • one book is supported by the table and wall.

  • Foresee Consequences:

    • the camera and the book will fall if the box moves.

Jia, Gallagher, Saxena and Chen


Reasoning with blocks support stability

Reasoning with Blocks, Support, & Stability

Input: RGB-D

Segmentation

Jia, Gallagher, Saxena and Chen


Reasoning with blocks support stability1

Reasoning with Blocks, Support, & Stability

Input: RGB-D

Blocks, Support, and Stability

Jia, Gallagher, Saxena and Chen


Reasoning with blocks support stability2

Reasoning with Blocks, Support, & Stability

Input: RGB-D

Final 3D representation

Jia, Gallagher, Saxena and Chen


Algorithms

Algorithms

Jia, Gallagher, Saxena and Chen


Overview

Overview

3D Block Fitting

Support and Stability

Input Segmentation*

Evaluate Energy Function

* "Indoor Segmentation and Support Inference from RGBD Images," N. Silberman et al. ECCV, 2012.

Jia, Gallagher, Saxena and Chen


Overview1

Overview

3D Block Fitting

Input Segmentation

3D Block Fitting

Support and Stability

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen


Single block fitting

Single Block Fitting

  • 3D orientated bounding box on depth data

  • Partially observed. Minimum volume may fail *

  • Minimum surface distance (Min-surf)

* "Fast oriented bounding box optimization on the rotation group SO(3, R)," C. Chang et al, ACM Transactions on Graphics, 2011.

Jia, Gallagher, Saxena and Chen


Overview2

Overview

3D Block Fitting

Input Segmentation

Support and Stability

Support and Stability

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen


Support and stability

Support and Stability

Support

Relations

Supporting

Area

Stability


Support relation

Support Relation

Surface On-top

Support

Partial On-top

Support

Side

Support

Jia, Gallagher, Saxena and Chen


3d based reasoning with blocks support and stability

Separate axis is perpendicular to y

Separate axis is parallel to y

Surface On-top

Support

Partial On-top

Support

Side

Support

Jia, Gallagher, Saxena and Chen


3d based reasoning with blocks support and stability

Surface On-top

Support

Partial On-top

Support

Side

Support

Jia, Gallagher, Saxena and Chen


From support to stability

From Support To Stability

  • Supporting Area

Jia, Gallagher, Saxena and Chen


From support to stability1

From Support To Stability

  • Supporting Area

  • Stability

Stable

Jia, Gallagher, Saxena and Chen


From support to stability2

From Support To Stability

  • Supporting Area

  • Stability

Stable

Unstable

Jia, Gallagher, Saxena and Chen


Overview3

Overview

3D Block Fitting

Input Segmentation

Support and Stability

Evaluate Energy Function

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen


Reasoning through an energy function

Reasoning Through an Energy Function

Segmentation Energy Function

Use Support Relations, Stability, Other Box-based/RGB-D info as features.

Better Segmentation

Smaller F(S)

Worse Segmentation

Larger F(S)

RGB-D

Jia, Gallagher, Saxena and Chen


Energy function single box potential

Energy Function: Single Box Potential

  • Features: minimum surface distance, visibility, single box stability, etc.

Worse

Box

Better

Box

Jia, Gallagher, Saxena and Chen


Energy function pairwise box potential

Energy Function: Pairwise Box Potential

  • Features: box intersection, support, supporting area distance etc.

Worse

Boundary

Better

Boundary

Jia, Gallagher, Saxena and Chen


3d based reasoning with blocks support and stability

1.4

……

……

2.3

Segmentation Energy Function:

Segmentation at one step

……

1.2

……

……

Jia, Gallagher, Saxena and Chen


Summary

Summary

3D Block Fitting

Support and Stability

Input Segmentation

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen


Experiments

Experiments

Jia, Gallagher, Saxena and Chen


Experiments1

Experiments:

  • Block dataset

  • Cornell Support Object dataset (SOD)

    • 300 RGB-D images with ground-truth segments and support relations

  • NYU-2 RGB-D dataset

Jia, Gallagher, Saxena and Chen


Experiment segmentation results

Experiment: Segmentation Results

  • Pixel-wise object segmentation accuracy:

Jia, Gallagher, Saxena and Chen


Experiment segmentation results1

Experiment: Segmentation Results

Input RGB-D images

Jia, Gallagher, Saxena and Chen


Experiments support inference

Experiments: Support Inference

  • Neighbor: object is supported by its neighbors

  • Stability: trim unnecessary support after reasoning

Jia, Gallagher, Saxena and Chen


3d based reasoning with blocks support and stability

Color Segmentation

D. Hoiem et al. ICCV, 2007;

P. Arbelaez et al. CVPR, 2012.

……

Semantic 3D Labeling

H. Koppula et. al. NIPS 2011.

Blocks world revisited

A. Gupta et all, ECCV, 2010.

Object Placement

Y. Jiang et al. IJRR, 2012.

Indoor Segmentation & Support

N. Silberman et al. ECCV 2012.

Jia, Gallagher, Saxena and Chen


Conclusion

Conclusion

  • 3D support and stability

    • Based on box representations

  • Object segmentation in 3D scene

    • Learning algorithm.

  • Future work

    • Non-uniform density

    • Semantic classification on blocks

    • Occluded supports

Jia, Gallagher, Saxena and Chen


3d based reasoning with blocks support and stability

3D-Based Reasoning with Blocks, Support, and Stability

Zhaoyin Jia, Andrew Gallagher, AshutoshSaxena, Tsuhan Chen

Cornell University

Thanks. Questions?


  • Login