- By
**keefe** - Follow User

- 126 Views
- Uploaded on

Download Presentation
## PowerPoint Slideshow about ' Segmentation from Examples' - keefe

**An Image/Link below is provided (as is) to download presentation**

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Algorithm for semantic segmentation Algorithm for semantic segmentation Algorithm for semantic segmentation

What is Segmentation from Examples?

- Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain
- Effective when making a semantic segmentation

Why to use Examples

- The example defines the granularity of the desired output
- Give us the ability to characterize meaningful parts in the image
- Using example allow us to use non-parametric model

The example defines the granularity of the desired output:

Desired Segmentation

Training image

Test Image

Induced Segmentation

Why to use Examples

- The example defines the granularity of the desired output
- Give us the ability to characterize meaningful parts in the image

Semantic Segmentation from an example

- We want to segment an image into semantically meaningful parts
- Required in various applications

Semantic Segmentation from an example

- We want to segment an image into semantically meaningful parts
- Required in various applications
- Problems:
- Meaningful parts are often too complex
- Semantic interpretation is highly subjective, depending on both the application, and the user

Semantic Segmentation from an example

- We want to segment an image into semantically meaningful parts
- Required in various applications
- Problems:
- Meaningful parts are often too complex
- Semantic interpretation is highly subjective, depending on both the application, and the user

Semantic Segmentation from an example

- So, How to achieve semantic segmentation
- Getting segmented training image(s) as input

Semantic Segmentation from an example

- So, How to achieve semantic segmentation
- Getting segmented training image(s) as input
- Using non-parametric representation

non-parametric model

Each semantic part is represented by a set of square patches

Semantic Segmentation from an example

- So, How to achieve semantic segmentation
- Getting segmented training image(s) as input
- Using non-parametric representation
- Over-segmenting the Test image into small fragments

Semantic Segmentation from an example

- So, How to achieve semantic segmentation
- Getting segmented training image(s) as input
- Using non-parametric representation
- Over-segmenting the Test image into small fragments
- Compute costs for fragment-label pairs

Semantic Segmentation from an example

- So, How to achieve semantic segmentation
- Getting segmented training image(s) as input
- Using non-parametric representation
- Over-segmenting the Test image into small fragments
- Compute costs for fragment-label pairs
- Graph-cuts multi-label optimization

Why do we need graph-cuts

- Graph-cuts optimization is used to label each fragment in a globally optimal manner

Classification scores

Patch sets

Classification

Graph-Cuts optimization

Fragments

Test image

Fragmentation

Result

Over segmenting

- Fragment: small arbitrarily-shaped and simply-connected pixel clusters
- We assume that small homogeneous regions always belong to the same semantic part

Over segmenting

- Fragment: small arbitrarily-shaped and simply connected pixel clusters
- We assume that small homogeneous regions always belong to the same semantic part
- Advantages:
- Enforces a locally coherent labeling
- Reduces the computational complexity

Graph-cuts multi-label optimization

- For each fragment we have k cost values, we need to determine which one is the optimal
- Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)

Algorithm for semantic segmentation

- Pixel labeling costs
- Fragmentation
- Fragment labeling costs
- Graph-cuts optimization

Algorithm for semantic segmentation

- Pixel labeling costs
- Fragmentation
- Fragment labeling costs
- Graph-cuts optimization

Pixel labeling costs

- Given Itrainand Ltrain
- Representing each label(segment) in Ltrain by a set of square patches
- We get k sets {Sl}l=1,…,k one for each label

Pixel labeling costs (cont.)

- Next, we define φ (p, l) for each (pixel,label) pair
- The cost of assigning label l to pixel pItest

p:pixel at Itest

l : label in Ltrain

ssd(P,P’) is the sum of squared distances between P,P’

M:mxmx3

P:mXm neighborhood centered at p

P’:mxmpatch

= min

P’Sl

Algorithm for semantic segmentation

- Pixel labeling costs
- Fragmentation
- Fragment labeling costs
- Graph-cuts optimization

Fragmentation

- We partition Itest into small,color-homogeneous regions using mean shift segmentation

Fragmentation

- We partition Itest into small,color-homogeneous regions using mean shift segmentation
- Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)

Fragmentation

- We partition Itest into small,color-homogeneous regions using mean shift segmentation
- Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)
- Fragment boundaries align with edges in the image

- Pixel labeling costs
- Fragmentation
- Fragment labeling costs
- Graph-cuts optimization

Fragment labeling costs

- Voting scheme in order to compute labeling costs of each fragment
- For each fragment fItest we pick a few representative pixels:

Rep(f)={pi f } i=I,…,Rf

Rf is proportional to |f|

Fragment labeling vs. pixel labeling

- Fragment labeling reducing complexity
- We have n=Assume |Rep(f)|= ,then we need to compute costs only for pixels

Fragment labeling vs. pixel labeling

Training seg.

Training image

Input image

- Enforces a locally coherent labeling

Fragment labeling

Pixel labeling

- Pixel labeling costs
- Fragmentation
- Fragment labeling costs
- Graph-cuts optimization

Graph-cuts optimization

After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i

fragment labeling costs. Costs range in the interval [0,1]

Graph-cuts optimization

- Now for each pixel pItestwe have a labeling cost
- We need to find Ltestthe globally optimal labeling
- Requirements:
- Minimizes the total labeling cost
- Consistent with presence (or absence) of edges

Graph-cuts optimization (cont.)

- For each pair of neighboring pixels we define:

Ψ(p,q,L(p),L(q))=

L(p),L(q) : labels assigned to p,q

: difference between pixels p,q (RGB euclidian distance)

Graph-cuts optimization (cont.)

- In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term E(L) (E(L) the value of a labeling scheme) in terms of fragments instead of pixels :

: cost of assigning fragment f to label L(f), weighted by the size of each fragment.

: neighboring fragments in Itest.

=

:controls trade-off between regions and boundaries

Intuition:

- Big value:

For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have and reduce the energy. Instead of 1- althout can be big too.

This mean we prefer continues regions and not edges.

Intuition:

- s value:

For pixels p,q even with similar colors the graph-cuts step won’t care about connecting them to different labels because of small value. Even with big value at1-multiplying still give us small E value.

Favors boundaries,holding out non-continues regions .

Graph-cuts optimization (cont.)

- Finally Ltest is determined by solving

Ltest=minLE(L)

Labeling after Graph-Cuts Optimization

Fragment labeling

Multi-label graph-cut

Colored nodes:labels

Squares : fragments

For each (fragment,label) pair we have an edge.

Edge weigh according to φ.

Edges between two squares weighed according to Ψ.

Multi-label graph-cut is NP-complete problem!

- Using Isolation Heuristic we can get an approximation of E(L)
- For 1ik construct a minimum weight isolating cut Ei for label Li .
- Determine h such that Eh has max weight.
- E=.
- Return E.

- Pixel labeling costs
- Fragmentation
- Fragment labeling costs
- Graph-cuts optimization

Bear results

- invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.
- We can’t separate multiple objects belonging to the same label (c).

Summary

- We saw that giving segmented example from the same domain of an image can effectively perform a semantic segmentation
- Using example also defines the granularity of the desired output
- determining whether an entity belongs to a particular semantic part is more easily done at the fragment level, than on a pixel-by-pixel basis
- Using graph-cuts with multi-label support can help making global optimization step for finding optimal labeling
- Only one parameter needed , controlling the trade-off between regions and boundaries

References

- Inducing Semantic Segmentation from an Example,

YaarSchnitman, YaronCaspi, Daniel Cohen-Or, andDaniLischinski.

- "Segmentation by Example“, Sameer Agarwal and Serge Belongie.
- Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system.
- Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239

Download Presentation

Connecting to Server..