Segmentation from examples
Download
1 / 59

Segmentation from Examples - PowerPoint PPT Presentation


  • 126 Views
  • Uploaded on

Segmentation from Examples. By: A’laa Kryeem Lecturer: Hagit Hel-Or. What is Segmentation from Examples ?. Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain Effective when making a semantic segmentation. Why to use Examples.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Segmentation from Examples' - keefe


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Segmentation from examples

Segmentation from Examples

By: A’laa Kryeem

Lecturer: Hagit Hel-Or


What is segmentation from examples
What is Segmentation from Examples?

  • Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain

  • Effective when making a semantic segmentation


Why to use examples
Why to use Examples

  • The example defines the granularity of the desired output

  • Give us the ability to characterize meaningful parts in the image

  • Using example allow us to use non-parametric model


The example defines the granularity of the desired output
The example defines the granularity of the desired output:

Desired Segmentation

Training image

Test Image

Induced Segmentation


Why to use examples1
Why to use Examples

  • The example defines the granularity of the desired output

  • Give us the ability to characterize meaningful parts in the image



Semantic segmentation from an example
Semantic Segmentation from an example

  • We want to segment an image into semantically meaningful parts

  • Required in various applications


Semantic segmentation from an example1
Semantic Segmentation from an example

  • We want to segment an image into semantically meaningful parts

  • Required in various applications

  • Problems:

    • Meaningful parts are often too complex

    • Semantic interpretation is highly subjective, depending on both the application, and the user



Semantic segmentation from an example2
Semantic Segmentation from an example

  • We want to segment an image into semantically meaningful parts

  • Required in various applications

  • Problems:

    • Meaningful parts are often too complex

    • Semantic interpretation is highly subjective, depending on both the application, and the user



Semantic segmentation from an example3
Semantic Segmentation from an example

  • So, How to achieve semantic segmentation

    • Getting segmented training image(s) as input



Semantic segmentation from an example4
Semantic Segmentation from an example

  • So, How to achieve semantic segmentation

    • Getting segmented training image(s) as input

    • Using non-parametric representation


Non parametric model
non-parametric model

Each semantic part is represented by a set of square patches


Semantic segmentation from an example5
Semantic Segmentation from an example

  • So, How to achieve semantic segmentation

    • Getting segmented training image(s) as input

    • Using non-parametric representation

    • Over-segmenting the Test image into small fragments



Semantic segmentation from an example6
Semantic Segmentation from an example

  • So, How to achieve semantic segmentation

    • Getting segmented training image(s) as input

    • Using non-parametric representation

    • Over-segmenting the Test image into small fragments

    • Compute costs for fragment-label pairs


Fragment label cost example
(fragment,label) cost example

?


Semantic segmentation from an example7
Semantic Segmentation from an example

  • So, How to achieve semantic segmentation

    • Getting segmented training image(s) as input

    • Using non-parametric representation

    • Over-segmenting the Test image into small fragments

    • Compute costs for fragment-label pairs

    • Graph-cuts multi-label optimization


Why do we need graph cuts
Why do we need graph-cuts

  • Graph-cuts optimization is used to label each fragment in a globally optimal manner


Training set

Classification scores

Patch sets

Classification

Graph-Cuts optimization

Fragments

Test image

Fragmentation

Result


Over segmenting
Over segmenting

  • Fragment: small arbitrarily-shaped and simply-connected pixel clusters

  • We assume that small homogeneous regions always belong to the same semantic part


Over segmenting1
Over segmenting

  • Fragment: small arbitrarily-shaped and simply connected pixel clusters

  • We assume that small homogeneous regions always belong to the same semantic part

  • Advantages:

    • Enforces a locally coherent labeling

    • Reduces the computational complexity


Graph cuts multi label optimization
Graph-cuts multi-label optimization

  • For each fragment we have k cost values, we need to determine which one is the optimal

  • Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)


Algorithm for semantic segmentation
Algorithm for semantic segmentation

  • Pixel labeling costs

  • Fragmentation

  • Fragment labeling costs

  • Graph-cuts optimization


Algorithm for semantic segmentation1
Algorithm for semantic segmentation

  • Pixel labeling costs

  • Fragmentation

  • Fragment labeling costs

  • Graph-cuts optimization


Pixel labeling costs
Pixel labeling costs

  • Given Itrainand Ltrain

    • Representing each label(segment) in Ltrain by a set of square patches

    • We get k sets {Sl}l=1,…,k one for each label


Pixel labeling costs cont
Pixel labeling costs (cont.)

  • Next, we define φ (p, l) for each (pixel,label) pair

    • The cost of assigning label l to pixel pItest

p:pixel at Itest

l : label in Ltrain

ssd(P,P’) is the sum of squared distances between P,P’

M:mxmx3

P:mXm neighborhood centered at p

P’:mxmpatch

= min

P’Sl


Algorithm for semantic segmentation2
Algorithm for semantic segmentation

  • Pixel labeling costs

  • Fragmentation

  • Fragment labeling costs

  • Graph-cuts optimization


Fragmentation
Fragmentation

  • We partition Itest into small,color-homogeneous regions using mean shift segmentation


Fragmentation1
Fragmentation

  • We partition Itest into small,color-homogeneous regions using mean shift segmentation

  • Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)


Fragmentation2
Fragmentation

  • We partition Itest into small,color-homogeneous regions using mean shift segmentation

  • Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)

  • Fragment boundaries align with edges in the image


Fragmentation cont
Fragmentation (cont.)

Random colorization

Detailed close-up


Algorithm for semantic segmentation3
Algorithm for semantic segmentation

  • Pixel labeling costs

  • Fragmentation

  • Fragment labeling costs

  • Graph-cuts optimization


Fragment labeling costs
Fragment labeling costs

  • Voting scheme in order to compute labeling costs of each fragment

  • For each fragment fItest we pick a few representative pixels:

    Rep(f)={pi f } i=I,…,Rf

    Rf is proportional to |f|


Fragment labeling costs cont
Fragment labeling costs (cont.)

  • We talked about the cost of assigning a pixel into a label

    Rep(f)}


Fragment labeling vs pixel labeling
Fragment labeling vs. pixel labeling

  • Fragment labeling reducing complexity

    • We have n=Assume |Rep(f)|= ,then we need to compute costs only for pixels


Fragment labeling vs pixel labeling1
Fragment labeling vs. pixel labeling

Training seg.

Training image

Input image

  • Enforces a locally coherent labeling

Fragment labeling

Pixel labeling


Algorithm for semantic segmentation4
Algorithm for semantic segmentation

  • Pixel labeling costs

  • Fragmentation

  • Fragment labeling costs

  • Graph-cuts optimization


Graph cuts optimization
Graph-cuts optimization

After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i

fragment labeling costs. Costs range in the interval [0,1]


Graph cuts optimization1
Graph-cuts optimization

  • Now for each pixel pItestwe have a labeling cost

  • We need to find Ltestthe globally optimal labeling

    • Requirements:

      • Minimizes the total labeling cost

      • Consistent with presence (or absence) of edges


Graph cuts optimization cont
Graph-cuts optimization (cont.)

  • For each pair of neighboring pixels we define:

    Ψ(p,q,L(p),L(q))=

L(p),L(q) : labels assigned to p,q

: difference between pixels p,q (RGB euclidian distance)


Graph cuts optimization cont1
Graph-cuts optimization (cont.)

  • In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term E(L) (E(L) the value of a labeling scheme) in terms of fragments instead of pixels :

    : cost of assigning fragment f to label L(f), weighted by the size of each fragment.

    : neighboring fragments in Itest.

    =

    :controls trade-off between regions and boundaries


Intuition
Intuition:

  • Big value:

    For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have and reduce the energy. Instead of 1- althout can be big too.

    This mean we prefer continues regions and not edges.


Intuition1
Intuition:

  • s value:

    For pixels p,q even with similar colors the graph-cuts step won’t care about connecting them to different labels because of small value. Even with big value at1-multiplying still give us small E value.

    Favors boundaries,holding out non-continues regions .


Graph cuts optimization cont2
Graph-cuts optimization (cont.)

  • Finally Ltest is determined by solving

    Ltest=minLE(L)

Labeling after Graph-Cuts Optimization

Fragment labeling


Multi label graph cut
Multi-label graph-cut

Colored nodes:labels

Squares : fragments

For each (fragment,label) pair we have an edge.

Edge weigh according to φ.

Edges between two squares weighed according to Ψ.


Multi label graph cut1
Multi-label graph-cut

Induced graph

Each fragment connected to a single label.


Multi label graph cut is np complete problem
Multi-label graph-cut is NP-complete problem!

  • Using Isolation Heuristic we can get an approximation of E(L)

  • For 1ik construct a minimum weight isolating cut Ei for label Li .

  • Determine h such that Eh has max weight.

  • E=.

  • Return E.


Effect
effect

Input image

Training image

Training segmentation

=5

=1

=0.1


Algorithm for semantic segmentation5
Algorithm for semantic segmentation

  • Pixel labeling costs

  • Fragmentation

  • Fragment labeling costs

  • Graph-cuts optimization


Algorithm results
Algorithm results

Training set

b

c

a


Bear results
Bear results

  • invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.

  • We can’t separate multiple objects belonging to the same label (c).


Algorithm results1
Algorithm results

b

a

d

c

Training set


Summary
Summary

  • We saw that giving segmented example from the same domain of an image can effectively perform a semantic segmentation

  • Using example also defines the granularity of the desired output

  • determining whether an entity belongs to a particular semantic part is more easily done at the fragment level, than on a pixel-by-pixel basis

  • Using graph-cuts with multi-label support can help making global optimization step for finding optimal labeling

  • Only one parameter needed , controlling the trade-off between regions and boundaries



References
References

  • Inducing Semantic Segmentation from an Example,

    YaarSchnitman, YaronCaspi, Daniel Cohen-Or, andDaniLischinski.

  • "Segmentation by Example“, Sameer Agarwal and Serge Belongie.

  • Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system.

  • Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239


ad