# 3D-Based Reasoning with Blocks, Support, and Stability - PowerPoint PPT Presentation

1 / 33

3D-Based Reasoning with Blocks, Support, and Stability. Zhaoyin Jia. School of Electrical and Computer Engineering Cornell University. Computer Vision with RGB-D. Pose Recognition J. Shotton et al. 2011; G. Girshick et al. 2013. Activity Detection

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

3D-Based Reasoning with Blocks, Support, and Stability

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

#### Presentation Transcript

3D-Based Reasoning with Blocks, Support, and Stability

Zhaoyin Jia

School of Electrical and Computer Engineering

Cornell University

### Computer Vision with RGB-D

Pose Recognition

J. Shotton et al. 2011; G. Girshick et al. 2013.

Activity Detection

J. Sung et al. 2012; H. Koppula et al. 2013.

Object Recognition

K. Lai et al. 2011; A. Janoch et al. 2011

3D Scene Labeling

H. Koppula, et al. 2011; N. Silberman et al 2011, 2012.

Jia, Gallagher, Saxena and Chen

### RGB-D Images

Jia, Gallagher, Saxena and Chen

### 3D Reasoning on RGB-D Images

• Free Space:

• objects can be placed in empty spaces.

• Physical Stability:

• one book is supported by the table and wall.

• Foresee Consequences:

• the camera and the book will fall if the box moves.

Jia, Gallagher, Saxena and Chen

### Reasoning with Blocks, Support, & Stability

Input: RGB-D

Segmentation

Jia, Gallagher, Saxena and Chen

### Reasoning with Blocks, Support, & Stability

Input: RGB-D

Blocks, Support, and Stability

Jia, Gallagher, Saxena and Chen

### Reasoning with Blocks, Support, & Stability

Input: RGB-D

Final 3D representation

Jia, Gallagher, Saxena and Chen

## Algorithms

Jia, Gallagher, Saxena and Chen

### Overview

3D Block Fitting

Support and Stability

Input Segmentation*

Evaluate Energy Function

* "Indoor Segmentation and Support Inference from RGBD Images," N. Silberman et al. ECCV, 2012.

Jia, Gallagher, Saxena and Chen

### Overview

3D Block Fitting

Input Segmentation

3D Block Fitting

Support and Stability

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen

### Single Block Fitting

• 3D orientated bounding box on depth data

• Partially observed. Minimum volume may fail *

• Minimum surface distance (Min-surf)

* "Fast oriented bounding box optimization on the rotation group SO(3, R)," C. Chang et al, ACM Transactions on Graphics, 2011.

Jia, Gallagher, Saxena and Chen

### Overview

3D Block Fitting

Input Segmentation

Support and Stability

Support and Stability

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen

Support

Relations

Supporting

Area

Stability

### Support Relation

Surface On-top

Support

Partial On-top

Support

Side

Support

Jia, Gallagher, Saxena and Chen

Separate axis is perpendicular to y

Separate axis is parallel to y

Surface On-top

Support

Partial On-top

Support

Side

Support

Jia, Gallagher, Saxena and Chen

Surface On-top

Support

Partial On-top

Support

Side

Support

Jia, Gallagher, Saxena and Chen

### From Support To Stability

• Supporting Area

Jia, Gallagher, Saxena and Chen

### From Support To Stability

• Supporting Area

• Stability

Stable

Jia, Gallagher, Saxena and Chen

### From Support To Stability

• Supporting Area

• Stability

Stable

Unstable

Jia, Gallagher, Saxena and Chen

### Overview

3D Block Fitting

Input Segmentation

Support and Stability

Evaluate Energy Function

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen

### Reasoning Through an Energy Function

Segmentation Energy Function

Use Support Relations, Stability, Other Box-based/RGB-D info as features.

Better Segmentation

Smaller F(S)

Worse Segmentation

Larger F(S)

RGB-D

Jia, Gallagher, Saxena and Chen

### Energy Function: Single Box Potential

• Features: minimum surface distance, visibility, single box stability, etc.

Worse

Box

Better

Box

Jia, Gallagher, Saxena and Chen

### Energy Function: Pairwise Box Potential

• Features: box intersection, support, supporting area distance etc.

Worse

Boundary

Better

Boundary

Jia, Gallagher, Saxena and Chen

1.4

……

……

2.3

Segmentation Energy Function:

Segmentation at one step

……

1.2

……

……

Jia, Gallagher, Saxena and Chen

### Summary

3D Block Fitting

Support and Stability

Input Segmentation

Evaluate Energy Function

Jia, Gallagher, Saxena and Chen

## Experiments

Jia, Gallagher, Saxena and Chen

### Experiments:

• Block dataset

• Cornell Support Object dataset (SOD)

• 300 RGB-D images with ground-truth segments and support relations

• NYU-2 RGB-D dataset

Jia, Gallagher, Saxena and Chen

### Experiment: Segmentation Results

• Pixel-wise object segmentation accuracy:

Jia, Gallagher, Saxena and Chen

### Experiment: Segmentation Results

Input RGB-D images

Jia, Gallagher, Saxena and Chen

### Experiments: Support Inference

• Neighbor: object is supported by its neighbors

• Stability: trim unnecessary support after reasoning

Jia, Gallagher, Saxena and Chen

Color Segmentation

D. Hoiem et al. ICCV, 2007;

P. Arbelaez et al. CVPR, 2012.

……

Semantic 3D Labeling

H. Koppula et. al. NIPS 2011.

Blocks world revisited

A. Gupta et all, ECCV, 2010.

Object Placement

Y. Jiang et al. IJRR, 2012.

Indoor Segmentation & Support

N. Silberman et al. ECCV 2012.

Jia, Gallagher, Saxena and Chen

### Conclusion

• 3D support and stability

• Based on box representations

• Object segmentation in 3D scene

• Learning algorithm.

• Future work

• Non-uniform density

• Semantic classification on blocks

• Occluded supports

Jia, Gallagher, Saxena and Chen

3D-Based Reasoning with Blocks, Support, and Stability

Zhaoyin Jia, Andrew Gallagher, AshutoshSaxena, Tsuhan Chen

Cornell University

Thanks. Questions?