Graph cut normalized cut segmentation
This presentation is the property of its rightful owner.
Sponsored Links
1 / 75

Graph-Cut / Normalized Cut segmentation PowerPoint PPT Presentation


  • 140 Views
  • Uploaded on
  • Presentation posted in: General

Graph-Cut / Normalized Cut segmentation. Jad silbak -University of Haifa . What we have, and what we want:. Most segmentations until now focusing on local features (K-Means). We would like to extract the global impression of an image .

Download Presentation

Graph-Cut / Normalized Cut segmentation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Graph cut normalized cut segmentation

Graph-Cut / Normalized Cut segmentation

Jadsilbak -University of Haifa


What we have and what we want

What we have, and what we want:

  • Most segmentations until now focusing on local features (K-Means).

  • We would like to extract the global impression of an image .

  • Since there are many possible partitions ofthe domain, how do we pick the right one?


Definitions and reminders

Definitions and reminders

  • An undirected graph is marked as where:

    • V - is a set one for each data element (e.g., pixel).

    • E - is a set of edges.

    • Given or (the edges between u &v), we define as the affinity between connected nodes.


Definitions and reminders1

Definitions and reminders

  • What are Graph-Cuts ?

    • Set of edges whose removal makes a graph disconnected.

    • It’s the partition of V into A,B such that and

  • What is a cut’s cost ?

    • The red edges are a graph-Cut ,the cut’s cost is the sum of all the values of the edges .

    • Sum(red)=4 = cut’s cost .


Min cut

Min-cut

  • A graph can be partitioned into two disjoint sets ,we define the partition cost as:

  • The bipartition of the graph which minimizes the cut value is called the Min-Cut .


Why not use regular min cut for the partition

Why not use regular min-cut for the partition?

  • Lets see an example :


Why not use regular min cut for the partition1

Why not use regular min-cut for the partition?

  • Everynode is connected to all the other nodes.

  • edge weights are inversely proportional to the distance between the two nodes.


Why not use regular min cut for the partition2

Why not use regular min-cut for the partition?

  • minimum cut criteria favors cutting small sets of isolated nodes in the graph .


The ideal cut is not the cut with the min weight

The ideal cut is not the cut with the min weight

  • The red cut is the cut we want , but the min-cut favors small partition.

  • Weight of cut is directly proportional to the number of edges in the cut.!

* Slide from Khurram Hassan-Shafique CAP5415 Slide credit: B. Freeman and A. Torralba Computer Vision 2003


The solution normalized cut ncut

The solution - normalized cut (Ncut)

  • Instead of looking at the value of total edge weight connecting the two partitions,

  • we compute the cut cost as a fraction of the total edge connections to all the nodes in the graph. We call this disassociation(התנערות) measure the normalized cut (Ncut).


Normalized cut ncut

Normalized cut (Ncut)

  • Cut(A,B) is sum of weights with one end in A and one end in B ,we want to minimize the cut cost.

  • Assoc(A,V) is sum of all edges with one end in A , we want to maximize the sum of all weights for every A,B element in the partition


Normalized cut ncut1

Normalized cut (Ncut)


Nassoc

Nassoc

  • In the same spirit, we can define a measure for total normalized association


Ncut and nassoc

Ncut and Nassoc

  • Using some mathematical manipulations we get that:

  • Minimizing the disassociation between the groups and maximizing the association within the groups, are in fact identical and can be satisfied simultaneously.


Example cut and ncut

Example :cut and Ncut

  • What is the min cut , and what is the min Ncut ?


Example cut and ncut1

Example :cut and Ncut

  • It easy to see the min-cut ,we have efficient algorithms for solving the min-cut problem.


Example cut and ncut2

Example :cut and Ncut

  • What we get is a larger cut , and a smaller Ncut .

  • is this the min-Ncut ??


Ncut complexity

Ncut complexity:

  • Unfortunately, minimizing normalized cut exactly is NP complete, even for the special case of graphs on grids.

  • However, we will show that, when we put the normalized cut problem in the real value domain, an approximate discrete solution can be found efficiently.


Adjacency matrix

Adjacency matrix

  • Let W be the adjacency matrix of the graph, where every is the similarity between i,j where .

e

a

2

1

b

2

1

c

2

d


Diagonal matrix

Diagonal matrix

  • Let D be the diagonal matrix with diagonal entries

  • ( D(i)=the sum of all edge with one end in )

e

a

2

1

b

2

1

c

2

d


Laplacian matrix

Laplacian matrix :

  • We define the thelaplacian matrix L :

a

e

2

1

b

2

1

c

2

d


Laplacian matrix properties

laplacian matrix properties:

  • The laplacian matrix properties :

    • All eigenvectors of L are perpendicular to each other (has a complete set of orthonormaleigen−vectors) .

    • L is symmetric positive semi-definite .

    • Has a N non-negative real-valued eigen-values ,the smallest eigen-values is always 0 with eigen- vector .

Vasileios Zografos [email protected] Klas Nordberg [email protected]


Indicator vector

Indicator vector

  • let x be an dimensional indicator vector, if node , and otherwise.

A

B

e

a

2

1

b

2

c

1

2

d


Normalized cut

Normalized cut

  • Then the min normalized cut cost can be written as:

  • Where y is an indicator vector that acts like x with one exception if then .


Y b definition and example

Y ,b definition and example:

  • We define b as .

A

B

a

e

2

1

b

2

c

1

2

d


Y constraints

Y -constraints

  • we have two constraints on y:

    1)

    • This can be seen as a constrain that forces all y indicator vectors to be perpendicular to each other and specifically the vector .

    • But we already know that the laplacian matrix has:

      • eigen vector of .

      • All eigen vectors are perpendicular to each other ,thus This constraints is automatically satisfied by the solution.


Y constraints1

Y -constraints

2)Y must be a discrete value one of two {1,-b} .

  • Satisfying this constraint is what makes the problem ‘np’.

  • If we relax this constraint to accept real values we are able to approximate the solution by solving the generalizedeigenvalue system .


Properties of the eigenvalue system

Properties of the eigenvalue system

  • The eigen vector will hopefully have similar values for nodes with high similarity( w(i,j) is high) .

  • The smallest eigenvector is always 0 , because we can have a partition of A=V and B={} thus Ncut(A,B)=0 .

  • Second smallest eigenvector is the real-valuedy thatminimizes ncut and is the solution for


What is clustering

What is Clustering?

  • Cluster: a collection of data objects

    • Similar to one another within the same cluster.

    • Dissimilar to the objects in other clusters.

    • The definition of clusters correspond to the definition of ncut.

A-cluster

B-cluster

a

e

2

1

b

2

c

1

2

d

Slide credit: Mario Haddad -Thanks


Spectral clustering

Spectral clustering

  • In order to find clusters that may be hard to find in the real domain (coordinates),such as non-convex data .

  • We use eigenvectors of matrices derived from the data to map the data to a low-dimensional space where the data is separated and can be easily clustered(we can use [K-Means] in the new space ).

  • Can be treated as graph partitioning problem without making specific assumptions on the form of the clusters (Ncut- is an example for spectral clustering).


Spectral clustering example

Spectral clusteringexample:

non-convex

convex

Vasileios Zografos [email protected] Klas Nordberg [email protected]


Why does this work

Why does this work?

  • Ideal Case

  • Why do the eigenvectors of the laplacian include cluster identification information?

https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CEQQFjAC&url=http%3A%2F%2Feniac.cs.qc.cuny.edu%2Fandrew%2Fgcml%2FLecture21.pptx&ei=RSM4U-qwKZDdsgbUh4DQDw&usg=AFQjCNElVvY0Mekord2Byc5qvb-8SlEuOg&sig2=IB53ooeEBUgrgPQRQ4nPew


Spectral clustering intuition

Spectral Clustering - Intuition

Slides Courtesy: Eric Xing, M. Hein & U.V. Luxburg


Spectral clustering intuition1

Spectral Clustering - Intuition

Slides Courtesy: Eric Xing, M. Hein & U.V. Luxburg


Spectral clustering intuition2

Spectral Clustering - Intuition

Slides Courtesy: Eric Xing, M. Hein & U.V. Luxburg


Why does this work1

Why does this work?

  • How does this eigenvector decomposition address this?

cluster assignment

  • https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CEQQFjAC&url=http%3A%2F%2Feniac.cs.qc.cuny.edu%2Fandrew%2Fgcml%2FLecture21.pptx&ei=RSM4U-qwKZDdsgbUh4DQDw&usg=AFQjCNElVvY0Mekord2Byc5qvb-8SlEuOg&sig2=IB53ooeEBUgrgPQRQ4nPew


Spectral clustering ncut

Spectral Clustering & Ncut

  • Ncut is type of spectral clustering, in witch we use the eigenvector as an indicator for the partition of the graph (data).

  • The eigenvalues represent the Ncut cost for each eigenvector, minimizing the Ncut means using a eigenvector with a corresponding min eigenvalue.

  • Since the smallest eigenvalue is “0” with a corresponding eigenvector (partitions the data to one cluster) the first eigenvector is not used .

  • The second eigenvector minimizes ncut and is the solution for :


Translating the eigenvector to partitions

Translating the eigenvector to partitions

  • What allowed us to approximate Ncut solution (which is ‘np’), is allowing the ‘y’ to take real value, but we still need a discrete value for the partition of the graph

a

e

2

1

b

2

c

1

2

d

Not discrete

second eigenvector


Translating the eigenvector to partitions1

Translating the eigenvector to partitions

  • We must choose a threshold so we can return to discrete value:

    • We can try random thresholds and choose the best.

    • We can check l evenly spaced possible splitting points and take the best ,(what gives the smallest Ncut).

a

e

2

Threshold T=‘0’

1

b

2

c

1

2

d


Some notes

Some notes:

  • We have seen how the second eigenvector can be used to bipartition of the graph, but what about the other eigenvector ??

  • The 3d smallest eigenvector can give a sub-partition for the partition we got from the 2d smallest eigenvector, and in general every eigenvector can sub-partition the result we got from the perverse eigenvector.


Some notes1

Some notes:

  • But do to the fact that every time we use an eigenvector to partition the graph, we may get errors from the conversion to discrete values, the error accumulates and the partition becomes less reliable the higher we go.

  • What do we do to farther partition the graph ?


Recursive two way ncut

Recursive Two-Way Ncut

  • Given compute the weight on each edge, and summarize the information into W and D.

  • Solve for eigenvectors with the smallest eigenvalues.

  • We use the eigenvector with the second smallest eigenvalue to bipartition the graph.


Recursive two way ncut1

Recursive Two-Way Ncut

  • Decide if the current partition should be subdivided by checking :

    • The stability(not continues) if the values of the of the eigenvector are continues from one intery to anther then, it may be the we are tiring to sub-partition the data were it is not needed.

    • (Ncut < T ) T is prespecified value we use to indicate when to stop farther partition this sub graph.

  • Recursively repartition the segmented parts if necessary.


Recursive two way ncut example

Recursive Two-Way Ncutexample :

Denis Hamad LASL – [email protected]


How does all this relate to images

How does all this relate to images?

  • Let G be a graph that represents the image I as fallows, for every pix in I there exists a node in G that represents it, Wij is the similarty between pix ‘i’ and ‘j’.


Example brightness images

Example: Brightness Images

  • How do we define similarity ?

  • Where is the spatial location of node i, and is a feature vector based on intensity.


Example brightness images1

Example: Brightness Images

  • We can change the parameters similar to bilateral filtering.


Examples

Examples :


Deferent definitions of similarities

Deferent definitions of similarities:

  • where is the spatial location of node i, and is a feature vector based on:

    • intensity

    • color

    • texture

  • In intensity the feature victor was of dimension ‘1’ but we can have a d-dim feature victor where the weight are the similarity between these vectors .


Color feature vector

Color feature vector

  • We have seen feature vector based on intensity.

  • What about feature vector based on colors

    • Where h, s, v are the HSV values, for color segmentation


Ncut with color

Ncut with color :


Texture feature vector

texture feature vector

  • Feature vector based on texture segmentation:

  • We can use spatial filter for where the DOOG filters at various scales and orientations .


Ncut with texture

Ncut with texture


Time complexity

Time complexity

  • Solving a standard eigenvalue problem for all eigenvectors takes , this is impractical !!!

  • Fortunately our graph partitioning has the following properties :

    • The graphs are often only locally connected.

    • only the top few eigenvectors are needed for graph partitioning.

    • the precision requirement for the eigenvectors is low.


Time complexity1

Time complexity

  • We can remove up to 90% of the total connections with each of the neighborhoods without affecting the eigenvector solution to the system.

  • Putting everything together, each of the matrix vector computations cost .

  • Recursive Two-Way Ncut may take


Summarizing

Summarizing:

  • We have seen why min-cut wasn’t reliable info for segmentation and why we need Ncut.

  • Although the Ncut problem is np we can approximate a solution using the generalized eigenvalue system.

  • How does Ncut graph relate to images and how do we define similarity, (intensity ,color and texture).

  • Recursive Two-Way Ncut algorithm .

  • Time complexity.


Interactive graph cuts

Interactive graph cuts

  • Automatic segmentation seems to never be perfect !

  • What we want is the ability to mark (impose) hard constraints ,by indicating certain pixels (seeds) that absolutely have to be part of object and certain pixels that absolutely have to be part of background .

  • red is the object

  • Blue is the background


Boundary and region

Boundary and Region

  • Boundary based methods

    • - based on local information (Derivatives kernels, Harris, Canny).

  • Region based methods

    • + statistics inside the region.

    • - often generates irregular boundaries and small Holes.

Daniel Heilper, CS Department, Haifa University


Boundary and region1

Boundary and Region

:Boundary

Region:

Daniel Heilper, CS Department, Haifa University


Cost function

cost function:

  • The cost function provides a soft constraint for segmentation and includes both region and boundary properties .

  • Let be a binary vector whose components can be either “obj” or “bkg” ,p the set of nods.


Cost function1

cost function:

  • Region Boundary

  • specifies a relative importance of the region Boundary properties term.


Intuition

:Intuition

  • The can be seen as the individual penalties for assigning pixel p to “object” and “background” .

    • For example may reflect on how the intensity of pixel p fits into a known intensity model (e.g. histogram) of the object and background

  • comprises the “boundary” properties of segmentation A , Coefficient interpreted as a penalty for a discontinuity between p and q.

    • is large when pixels p and q are similar .

    • Costs may be based on local intensity gradient, Laplacian zero-crossing.


Examples1

Examples:

  • ->

Figure 2. Synthetic Gestalt example. The segmentation

results in (b-d)

are shown for various

Levels of relative importance of “region”

versus “boundary” in (1). Note that the result

in (b) corresponds to a wide range of .


Implementation

Implementation

The general work :

we create a graph with two terminals.

The edge weights reflect the parameters in the regional and the boundary terms of the cost function,

as well as the known positions of seeds in the image.

The seeds are O = {v} and B = {p}

Proceedings of “Internation Conference on Computer Vision”, Vancouver, Canada, July 2001


Interactive graph cuts in medical images

interactive graph cuts in medical images


Credits references

Credits & References

  • J. Shi and J.Malik, “Normalized Cuts and Image Segmentation,” Proc. CVPR 1997.also IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(8), 888-905, August 2000. 

  • Boykov, Y., Jolly, M., " Interactive graph cuts for optimal boundary and regionsegmentation of objects in N-D images." In: International Conference on Computer Vision, Vancouver , BC. (2001) 105–112.

  • PAMI 2000! Slide credit: S. Lazebnik.

  • Interactive Image Segmentation FahimMannan (260 266 294) .

  • Slides Courtesy: Eric Xing, M. Hein & U.V. Luxburg.

  • Daniel Heilper, CS Department, Haifa University.

  • https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&ved=0CEQQFjAC&url=http%3A%2F%2Feniac.cs.qc.cuny.edu%2Fandrew%2Fgcml%2FLecture21.pptx&ei=RSM4U-qwKZDdsgbUh4DQDw&usg=AFQjCNElVvY0Mekord2Byc5qvb-8SlEuOg&sig2=IB53ooeEBUgrgPQRQ4nPew


Thank you all

Thank you all


  • Login