Semantic texton forests for image categorization and segmentation
Download
1 / 24

Semantic Texton Forests for Image Categorization and Segmentation - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Semantic Texton Forests for Image Categorization and Segmentation. We would like to thank Amnon Drory for this deck. הבהרה: החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת. . Semantic Texton Forests. Input:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Semantic Texton Forests for Image Categorization and Segmentation' - nikkos


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Semantic texton forests for image categorization and segmentation

Semantic Texton Forests for Image Categorization and Segmentation

We would like to thank Amnon Drory

for this deck

הבהרה:

החומר המחייב הוא החומר הנלמד בכיתה ולא זה המופיע / לא מופיע במצגת.


Semantic texton forests
Semantic SegmentationTexton Forests

  • Input:

  • Training: Images with pixel level ground truth classification

MSRC 21 Database


Semantic texton forests1
Semantic SegmentationTexton Forests

  • Input:

  • Training: Images with pixel level ground truth classification.

  • Testing: Images

  • Output:

    • A classification of each pixel in the test images to an object class.


Main mathematical tools
Main Mathematical Tools Segmentation

  • Conditional Random Fields (CRF)

  • Textons( Convolution with Filter Banks )

  • K-Means

  • Joint Boost

  • Randomized Decision Forests


Decision trees
Decision Trees Segmentation


Decision trees1
Decision Trees Segmentation


Decision rules split functions
Decision Rules (Split SegmentationFunctions)

  • Choose one or two pixels near pixel of interest

  • Calculate simple function of their values:

    • Difference

    • Sum

    • Absolute Difference

    • Or just the pixel valuere To a hreshold

  • Motivation: very small number of computer operations. Easy to implement on GPU.


Training the tree
Training the tree Segmentation

  • For each node:

  • Randomly generate a few decision rules

  • Choose the one that maximally improves the ability of the tree to separate between classes. (E(I) – entropy)

  • Stop when tree reached pre-defined depth , or when no further improvement in classification can be achieved.


Decision forests
Decision Forests Segmentation

For added strength, create several trees instead of just one.

Each tree is trained using a different subset of the training data.


Classifying at test time
Classifying Segmentationat test time

0.001

0.04

0.05

0.23

0.01

For each pixel in the test image:

Apply the segmentation forest – marking a path in each tree (yellow).

Each leaf is associated with a histogram of classes.

Average the histograms from all tree, achieving a vector of probabilities for this pixel belonging to each class:


Classifying at test time1
Classifying Segmentationat test time

0.001

0.04

0.05

0.23

0.01

The probability vectors derived from the Decision Forests can be used to classify pixels to classes, by assigning to each pixel the label that is most likely. The results are very noisy.


Paths on trees represent texture
Paths on trees represent Texture Segmentation

1

84

85

86

2

3

4

87

7

6

5

91

9

10

8

11

17

The texture of an area around a pixel can be represented by a vector comprised of all the nodes in the decision forest that belong to paths traversed when applying the forest to this pixel. In the above example, this would be the vector:

( 1, 3, 6, 10, 17, … , 84, 85, 87, 91 )

This vector is called a Semantic Texton.


Example of semantic textons
Example Segmentationof Semantic Textons

A visualization of leaf nodes from

one tree (distance d = 21 pixels). Each patch is the average of all patches in the training images assigned to a particular leaf node. Features evident include color, horizontal, vertical and diagonal edges, blobs, ridges and corners.


Example of semantic textons1
Example Segmentationof Semantic Textons


Second randomized decision forest
Second Randomized Decision Forest Segmentation

  • The split functions at the nodes are based on the Texture-Layout filter:

  • Two types of Texture-Layout filters are used:

  • Count the occurrence of a certain node in the semantic textonsof pixels in a rectangle.

,

(

)


Second randomized decision forest1
Second Randomized Decision Forest Segmentation

  • The split functions at the nodes are based on the Texture-Layout filter:

  • Two types of Texture-Layout filters are used:

  • Count the occurrence of a certain node N in the semantic textons of pixels in a rectangle.

  • Sum the probabilities of belonging to a certain class K for all pixels in a rectangle. This is semantic context.

,

(

)


Stf results
STF Segmentation- Results

Overall Accuracy: 72%

Though less aesthetic, these results are quantitatively

almost as good as those of TextonBoost.


Stf summary
STF - Summary Segmentation

  • To speed up calculation, this algorithm uses Radomized Decision Forests instead of other mathemaical tools used in TextonBoost.

  • One RDF is used to calculate texture.

  • A second RDF is used to classifiy pixels to object classes.

  • The results are almost as good (quantitatively) as those of TextonBoost.

  • The algorithm is much quicker than TextonBoost.


Real time human pose recognition in parts from single depth images

Real-Time SegmentationHuman Pose Recognition in Parts from Single Depth Images

J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, A. Blake


Overview
Overview Segmentation


Classification
Classification Segmentation

  • Classify each pixel to one of 31 body part classes:

    • Left Upper / Right Upper / Left Lower / Right Lower head

    • neck

    • Left / Right shoulder

    • Left Upper / Right Upper / Left Lower / Right Lower arm

    • Left /Right elbow

    • Left /Right wrist

    • Left /Right hand

    • Left Upper / Right Upper / Left Lower / Right Lower torso

    • Left Upper / Right Upper / Left Lower / Right Lower leg

    • Left /Right knee

    • Left /Right ankle

    • Left /Right foot


Randomized decision forests
Randomized Decision Forests Segmentation

  • The Randomized Decision Forests are very deep (depth = 20).

  • => A very strong ability to classify correctly on the Training Set.

  • => A risk of over fitting. This risk is averted by using a very (very) large training set, containing examples of many poses we wish to recognize. Most of this training set is artificially generated.


Decision rules
Decision Rules Segmentation


Classification body part locations
Classification Segmentation→ body part locations

  • Separately for each of 31 Body Part Classes:

    • Create Probability map

    • Project to 3D space

    • Find modes ( using mean-shift )

    • The modes (with a little post processing ) are the suggested body part locations (the output of this algorithm).


ad