segmentation from examples
Download
Skip this Video
Download Presentation
Segmentation from Examples

Loading in 2 Seconds...

play fullscreen
1 / 59

Segmentation from Examples - PowerPoint PPT Presentation


  • 126 Views
  • Uploaded on

Segmentation from Examples. By: A’laa Kryeem Lecturer: Hagit Hel-Or. What is Segmentation from Examples ?. Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain Effective when making a semantic segmentation. Why to use Examples.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Segmentation from Examples' - keefe


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
segmentation from examples

Segmentation from Examples

By: A’laa Kryeem

Lecturer: Hagit Hel-Or

what is segmentation from examples
What is Segmentation from Examples?
  • Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain
  • Effective when making a semantic segmentation
why to use examples
Why to use Examples
  • The example defines the granularity of the desired output
  • Give us the ability to characterize meaningful parts in the image
  • Using example allow us to use non-parametric model
the example defines the granularity of the desired output
The example defines the granularity of the desired output:

Desired Segmentation

Training image

Test Image

Induced Segmentation

why to use examples1
Why to use Examples
  • The example defines the granularity of the desired output
  • Give us the ability to characterize meaningful parts in the image
semantic segmentation from an example
Semantic Segmentation from an example
  • We want to segment an image into semantically meaningful parts
  • Required in various applications
semantic segmentation from an example1
Semantic Segmentation from an example
  • We want to segment an image into semantically meaningful parts
  • Required in various applications
  • Problems:
    • Meaningful parts are often too complex
    • Semantic interpretation is highly subjective, depending on both the application, and the user
semantic segmentation from an example2
Semantic Segmentation from an example
  • We want to segment an image into semantically meaningful parts
  • Required in various applications
  • Problems:
    • Meaningful parts are often too complex
    • Semantic interpretation is highly subjective, depending on both the application, and the user
semantic segmentation from an example3
Semantic Segmentation from an example
  • So, How to achieve semantic segmentation
    • Getting segmented training image(s) as input
semantic segmentation from an example4
Semantic Segmentation from an example
  • So, How to achieve semantic segmentation
    • Getting segmented training image(s) as input
    • Using non-parametric representation
non parametric model
non-parametric model

Each semantic part is represented by a set of square patches

semantic segmentation from an example5
Semantic Segmentation from an example
  • So, How to achieve semantic segmentation
    • Getting segmented training image(s) as input
    • Using non-parametric representation
    • Over-segmenting the Test image into small fragments
semantic segmentation from an example6
Semantic Segmentation from an example
  • So, How to achieve semantic segmentation
    • Getting segmented training image(s) as input
    • Using non-parametric representation
    • Over-segmenting the Test image into small fragments
    • Compute costs for fragment-label pairs
semantic segmentation from an example7
Semantic Segmentation from an example
  • So, How to achieve semantic segmentation
    • Getting segmented training image(s) as input
    • Using non-parametric representation
    • Over-segmenting the Test image into small fragments
    • Compute costs for fragment-label pairs
    • Graph-cuts multi-label optimization
why do we need graph cuts
Why do we need graph-cuts
  • Graph-cuts optimization is used to label each fragment in a globally optimal manner
slide22

Training set

Classification scores

Patch sets

Classification

Graph-Cuts optimization

Fragments

Test image

Fragmentation

Result

over segmenting
Over segmenting
  • Fragment: small arbitrarily-shaped and simply-connected pixel clusters
  • We assume that small homogeneous regions always belong to the same semantic part
over segmenting1
Over segmenting
  • Fragment: small arbitrarily-shaped and simply connected pixel clusters
  • We assume that small homogeneous regions always belong to the same semantic part
  • Advantages:
    • Enforces a locally coherent labeling
    • Reduces the computational complexity
graph cuts multi label optimization
Graph-cuts multi-label optimization
  • For each fragment we have k cost values, we need to determine which one is the optimal
  • Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)
algorithm for semantic segmentation
Algorithm for semantic segmentation
  • Pixel labeling costs
  • Fragmentation
  • Fragment labeling costs
  • Graph-cuts optimization
algorithm for semantic segmentation1
Algorithm for semantic segmentation
  • Pixel labeling costs
  • Fragmentation
  • Fragment labeling costs
  • Graph-cuts optimization
pixel labeling costs
Pixel labeling costs
  • Given Itrainand Ltrain
    • Representing each label(segment) in Ltrain by a set of square patches
    • We get k sets {Sl}l=1,…,k one for each label
pixel labeling costs cont
Pixel labeling costs (cont.)
  • Next, we define φ (p, l) for each (pixel,label) pair
    • The cost of assigning label l to pixel pItest

p:pixel at Itest

l : label in Ltrain

ssd(P,P’) is the sum of squared distances between P,P’

M:mxmx3

P:mXm neighborhood centered at p

P’:mxmpatch

= min

P’Sl

algorithm for semantic segmentation2
Algorithm for semantic segmentation
  • Pixel labeling costs
  • Fragmentation
  • Fragment labeling costs
  • Graph-cuts optimization
fragmentation
Fragmentation
  • We partition Itest into small,color-homogeneous regions using mean shift segmentation
fragmentation1
Fragmentation
  • We partition Itest into small,color-homogeneous regions using mean shift segmentation
  • Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)
fragmentation2
Fragmentation
  • We partition Itest into small,color-homogeneous regions using mean shift segmentation
  • Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)
  • Fragment boundaries align with edges in the image
fragmentation cont
Fragmentation (cont.)

Random colorization

Detailed close-up

algorithm for semantic segmentation3
Algorithm for semantic segmentation
  • Pixel labeling costs
  • Fragmentation
  • Fragment labeling costs
  • Graph-cuts optimization
fragment labeling costs
Fragment labeling costs
  • Voting scheme in order to compute labeling costs of each fragment
  • For each fragment fItest we pick a few representative pixels:

Rep(f)={pi f } i=I,…,Rf

Rf is proportional to |f|

fragment labeling costs cont
Fragment labeling costs (cont.)
  • We talked about the cost of assigning a pixel into a label

Rep(f)}

fragment labeling vs pixel labeling
Fragment labeling vs. pixel labeling
  • Fragment labeling reducing complexity
    • We have n=Assume |Rep(f)|= ,then we need to compute costs only for pixels
fragment labeling vs pixel labeling1
Fragment labeling vs. pixel labeling

Training seg.

Training image

Input image

  • Enforces a locally coherent labeling

Fragment labeling

Pixel labeling

algorithm for semantic segmentation4
Algorithm for semantic segmentation
  • Pixel labeling costs
  • Fragmentation
  • Fragment labeling costs
  • Graph-cuts optimization
graph cuts optimization
Graph-cuts optimization

After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i

fragment labeling costs. Costs range in the interval [0,1]

graph cuts optimization1
Graph-cuts optimization
  • Now for each pixel pItestwe have a labeling cost
  • We need to find Ltestthe globally optimal labeling
    • Requirements:
      • Minimizes the total labeling cost
      • Consistent with presence (or absence) of edges
graph cuts optimization cont
Graph-cuts optimization (cont.)
  • For each pair of neighboring pixels we define:

Ψ(p,q,L(p),L(q))=

L(p),L(q) : labels assigned to p,q

: difference between pixels p,q (RGB euclidian distance)

graph cuts optimization cont1
Graph-cuts optimization (cont.)
  • In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term E(L) (E(L) the value of a labeling scheme) in terms of fragments instead of pixels :

: cost of assigning fragment f to label L(f), weighted by the size of each fragment.

: neighboring fragments in Itest.

=

:controls trade-off between regions and boundaries

intuition
Intuition:
  • Big value:

For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have and reduce the energy. Instead of 1- althout can be big too.

This mean we prefer continues regions and not edges.

intuition1
Intuition:
  • s value:

For pixels p,q even with similar colors the graph-cuts step won’t care about connecting them to different labels because of small value. Even with big value at1-multiplying still give us small E value.

Favors boundaries,holding out non-continues regions .

graph cuts optimization cont2
Graph-cuts optimization (cont.)
  • Finally Ltest is determined by solving

Ltest=minLE(L)

Labeling after Graph-Cuts Optimization

Fragment labeling

multi label graph cut
Multi-label graph-cut

Colored nodes:labels

Squares : fragments

For each (fragment,label) pair we have an edge.

Edge weigh according to φ.

Edges between two squares weighed according to Ψ.

multi label graph cut1
Multi-label graph-cut

Induced graph

Each fragment connected to a single label.

multi label graph cut is np complete problem
Multi-label graph-cut is NP-complete problem!
  • Using Isolation Heuristic we can get an approximation of E(L)
  • For 1ik construct a minimum weight isolating cut Ei for label Li .
  • Determine h such that Eh has max weight.
  • E=.
  • Return E.
effect
effect

Input image

Training image

Training segmentation

=5

=1

=0.1

algorithm for semantic segmentation5
Algorithm for semantic segmentation
  • Pixel labeling costs
  • Fragmentation
  • Fragment labeling costs
  • Graph-cuts optimization
algorithm results
Algorithm results

Training set

b

c

a

bear results
Bear results
  • invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.
  • We can’t separate multiple objects belonging to the same label (c).
algorithm results1
Algorithm results

b

a

d

c

Training set

summary
Summary
  • We saw that giving segmented example from the same domain of an image can effectively perform a semantic segmentation
  • Using example also defines the granularity of the desired output
  • determining whether an entity belongs to a particular semantic part is more easily done at the fragment level, than on a pixel-by-pixel basis
  • Using graph-cuts with multi-label support can help making global optimization step for finding optimal labeling
  • Only one parameter needed , controlling the trade-off between regions and boundaries
references
References
  • Inducing Semantic Segmentation from an Example,

YaarSchnitman, YaronCaspi, Daniel Cohen-Or, andDaniLischinski.

  • "Segmentation by Example“, Sameer Agarwal and Serge Belongie.
  • Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system.
  • Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239
ad