1 / 70

Graph-Cuts

Graph-Cuts. Michael Bleyer LVA Stereo Vision. What happened last time? (1). We have defined an energy function to measure the quality of a disparity map D : where m ( p , d p ) computes color dissimilarity for matching pixel p at disparity d p

iliana-burris
Download Presentation

Graph-Cuts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Graph-Cuts Michael Bleyer LVA Stereo Vision

  2. What happened last time? (1) • We have defined an energy function to measure the quality of a disparity map D: where • m(p,dp) computes color dissimilarity for matching pixel p at disparity dp • N denotes all spatial neighbors in 4-connectivity • s() is the smoothness function. We use the Potts model: • This energy function is important for many computer vision problems. 0 if dp = dq P otherwise.

  3. What happened last time? (2) • Smoothness interactions define a graph known as 4-connected grid. • Computing the energy optimum on the 4-connected grid is an np-complete problem. • We have learned about dynamic programming: • Computes exact energy optimum • Requires the graph to be a tree=> We had to remove smoothness interactions 4-Connected Grid

  4. What is Going to Happen Today? • Just one point on the agenda: • Graph-Cuts

  5. What is Graph-Cuts? • Powerful optimization method. • Finds strong local minima of our np-complete energy function. • Graph-cuts have been around in computer vision for quite some time (e.g. [Roy,ICCV98]). • I will speak about modern graph-cuts, i.e. move making algorithms

  6. Move Making Algorithms β γ • We are given a labeled image as input. • (In our case, the image is labeled with disparity values, i.e. label α can for example mean a disparity of 10 pixels.) • We want to modify the assignment of pixels to labels to obtain a better solution, i.e. one of lower energy. • An operation that changes labels is called a move. • We will learn about 3 types of moves: • αβ-swap • α-expansion • fusion move α Current labeling Move New labeling (preferably of lower energy than current labeling)

  7. αβ-Swap [Boykov,PAMI01] β γ α • Select two labels: α and β. • A pixel that is assigned to α in the current labeling can either: • switch its label to β or • keep its old label α in the new labeling. • Analogously, a pixel that is currently assigned to β can either: • switch its label to α or • keep its old label β in the new labeling. • Simply spoken: • Some pixels that had the label α are now assigned to β. • Some pixels that had the label β are now assigned to α. Current labeling β γ α • One possible labeling after αβ-swap

  8. α-Expansion [Boykov,PAMI01] β γ α • Select one label: α. • Any pixel can either • switch its label to α or • keep its old label. • More global than αβ-swap: • All pixels can change their labels simultaneously. • In experiments, α-expansion moves typically outperform αβ-swaps. • We will therefore concentrate on α-expansions. Current labeling β γ α • One possible labeling after α-expansion

  9. The Key Problem β α γ α-exp 2 α-exp 1 α β γ • There is an extremely large number of possible α-expansions. • The key challenge is to find the “best” α-expansion, i.e. the one that leads to the largest decrease of our energy. • Good news: • For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph. α β β α-exp 3 α-exp 4 γ γ Current labeling α α

  10. E = 2000 E = 1000 The Key Problem β α γ E = 1000 α-exp 2 α-exp 1 α β γ • There is an extremely large number of possible α-expansions. • The key challenge is to find the “best” α-expansion, i.e. the one that leads to the largest decrease of our energy. • Good news: • For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph. α β β α-exp 3 α-exp 4 γ γ Current labeling α α We should take this one E = 500 E = 750

  11. The Key Problem β α γ α-exp 2 α-exp 1 α β γ • There is an extremely large number of possible α-expansions. • The key challenge is to find the “best” α-expansion, i.e. the one that leads to the largest decrease of our energy. • Good news: • For our energy function, we can solve this problem in an exact and fast way via solving a min-cut problem in a graph. α β β α-exp 3 α-exp 4 γ γ Current labeling α α

  12. Iterative Algorithm – α-Expansion • Let us for now assume that we know how to compute the optimal α-expansion. • We can incorporate the α-expansion as follows. Iterative Algorithm: • Start with an arbitrary labeling f. • Loop (e.g. 3 times) • For each allowed label α: • Find f* = argmin E(f’) among f’ within one α-expansion of f • f := f* • Comment: • Note that we compute the optimal α-expansion. • Therefore, the energy will either decrease after α-expansion or stay the same (not changing the labeling at all is a feasible α-expansion). • The algorithm will in any case converge to a (strong) local energy optimum.

  13. Iterative Algorithm – Example Video (α-expansions for stereo matching)

  14. Computing the Optimal α-Expansion • There are 3 things you have to do to find the optimal α-expansion via graph-cuts: • Write your energy as a pseudo-boolean function • Construct a graph that represents your boolean function • Compute the Minimum Cut in this graph • These steps are discussed in the following.

  15. Writing the Energy as a Pseudo-Boolean Function (1) • We associate a boolean variable xp with each pixel p where: • xp = 0 means that pixel p keeps its old label after α-expansion • xp = 1 means that pixel p takes label α after α-expansion • For example, if this is the current labeling:then x = leads to the label configuration:after α-expansion. • We can represent all possible α-expansions by the boolean variables x. β β β γ γ 1 1 0 0 1 α α β γ α

  16. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling:

  17. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: Potts model (Impose penalty P if p and q have different labels) Dissimilarity function

  18. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:

  19. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:

  20. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:

  21. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq: We call these terms unary terms, since they depend on one variable. We call this term a pairwise term, since it depends on two variables.

  22. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where:

  23. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where:

  24. Writing the Energy as a Pseudo-Boolean Function (2) β β • Let us assume we have two pixels p and q. • Both pixels are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where: β β β α α β α α

  25. Writing the Energy as a Pseudo-Boolean Function (2) We have to find the settings of binary variables xp and xq that minimize the energy. This comes next. β β • Let us assume we have two pixels p and q. • Both pixel are assigned to label β in the current labeling: • Recall our energy function: • We can write our energy as a function of binary variables xp and xq:where: β β β α α β α α

  26. The Min-Cut Problem source 2 9 • We have two dedicated nodes, the source and the sink. 2 p q 1 5 4 sink ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

  27. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where 2 p q 1 5 4 T sink ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

  28. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where • The cut consists of all edges that lead from S to T. 2 p q 1 5 4 T sink ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

  29. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where • The cut consists of all edges that lead from S to T. • The costs of a cut are the sum of weights of these edges. 2 p q 1 5 4 T sink Costs: 5 + 2 + 9 = 16 ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

  30. The Min-Cut Problem S source 2 9 • We have two dedicated nodes, the source and the sink. • We partition the graph into two sets S and T where • The cut consists of all edges that lead from S to T. • The costs of a cut are the sum of weights of these edges. • The minimum cut is the cut of minimum costs among all possible cuts. 2 p q 1 5 4 T sink Costs: 2 + 1 + 4 = 7 ExampletakenfromPushmeet Kohli’s ICCV09 tutorial

  31. The Min-Cut Problem • The min-cut problem has been extensively studied in graph theory. • There exists a variety of algorithms that • Can find the exact solution • Are computationally very fast. • Side notes: • The min-cut problem and the max-flow problem are dual problems: => Solving min-cut also gives the solution for max-flow and vice versa. • Max-flow and min-cut are therefore often used synonymously. • If you are interested in algorithms for computing min-cut/max-flow: • Read [Boykov,PAMI04]

  32. The Min-Cut Problem Nice, but does this help us to optimize our pseudo-boolean function? • The min-cut problem has been extensively studied in graph theory. • There exists a variety of algorithms that • Can find the exact solution • Are computationally very fast. • Side notes: • The min-cut problem and the max-flow problem are dual problems: => Solving min-cut also gives the solution for max-flow and vice versa. • Max-flow and min-cut are therefore often used synonymously. • If you are interested in algorithms for computing min-cut/max-flow: • Read [Boykov,PAMI04]

  33. Optimization of our Pseudo-Boolean Function source • We insert a node for each pixel. p q sink

  34. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. => xp = 0 p q sink

  35. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 => xp = 0 p q => xq = 1 T sink

  36. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 • We adjust the edges so that the costs of the cut are equal to the energy of our binary variables x. => xp = 0 p q => xq = 1 T sink The costs of this cut have to be equal to the energy of xp = 0 and xq= 1.

  37. Optimization of our Pseudo-Boolean Function S source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 • We adjust the edges so that the costs of the cut are equal to the energy of our binary variables x. • The minimum cut therefore also represents the minimum of our energy. => xp = 0 p q => xq = 1 T sink The costs of this cut have to be equal to the energy of xp = 0 and xq= 1.

  38. Optimization of our Pseudo-Boolean Function S How can we do this for our example? source • We insert a node for each pixel. • If a node p is member of S after the cut, then xp = 0. • If p is member of T, then xp = 1 • We adjust the edges so that the costs of the cut are equal to the energy of our binary variables x. • The minimum cut therefore also represents the minimum of our energy. => xp = 0 p q => xq = 1 T sink The costs of this cut have to be equal to the energy of xp = 0 and xq= 1.

  39. Optimization of our Pseudo-Boolean Function source • Our unary terms: • Our pairwise term: p q sink

  40. Optimization of our Pseudo-Boolean Function source m(p,α) • Our unary terms: • Our pairwise term: p q m(p,β) sink

  41. Optimization of our Pseudo-Boolean Function source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: p q m(q,β) m(p,β) sink

  42. Optimization of our Pseudo-Boolean Function source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) sink

  43. Optimization of our Pseudo-Boolean Function Let us check whether this graph really represents our energy. source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) sink

  44. Optimization of our Pseudo-Boolean Function => xp = 0, xq = 0 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(0)+Eq(0)+Ep,q(0,0) = m(p,β)+m(q,β)+0 Cut Costs:C = m(p,β)+m(q,β)

  45. Optimization of our Pseudo-Boolean Function => xp = 0, xq = 1 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(0)+Eq(1)+Ep,q(0,1) = m(p,β)+m(q,α)+P Cut Costs:C = m(p,β)+m(q,α)+P

  46. Optimization of our Pseudo-Boolean Function => xp = 1, xq = 0 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(1)+Eq(0)+Ep,q(1,0) = m(p,α)+m(q,β)+P Cut Costs:C = m(p,α)+m(q,β)+P

  47. Optimization of our Pseudo-Boolean Function => xp = 1, xq = 1 S source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(1)+Eq(1)+Ep,q(1,1) = m(p,α)+m(q,α)+0 Cut Costs:C = m(p,α)+m(q,α)

  48. Optimization of our Pseudo-Boolean Function => xp = 1, xq = 1 S We have shown that the graph represents our energy. source m(q,α) m(p,α) • Our unary terms: • Our pairwise term: P p q P m(q,β) m(p,β) T sink Energy: E(0,1) = Ep(1)+Eq(1)+Ep,q(1,1) = m(p,α)+m(q,α)+0 Cut Costs:C = m(p,α)+m(q,α)

  49. What Energy Function Can be Optimized via Graph-Cuts? • Not every boolean energy function can be represented by a graph! • The pairwise terms have to fulfill the following constraint [Kolmogorov,PAMI04]: • In our example, this has been the case: • If there is at least one pairwise term in the boolean energy function that violates this constraint, the energy is said to be non-submodular. • Otherwise, it is called submodular. • Optimizing non-submodular energies is an np-complete problem.=> Computing the optimal α-expansion becomes very difficult.  (but not impossible)

  50. Max-Flow/Min-Cut Library • “I would like to use graph-cuts, but I do not want to mess around with graphs.” • Good news: • You don’t have to. • It is sufficient to define your energy as a pseudo-boolean function. • You can then download the Max-Flow/Min-Cut library from http://www.cs.ucl.ac.uk/staff/V.Kolmogorov/software.html • The library will: • Construct the graph that represents your boolean function • Compute the min-cut • Provide you the optimal labeling • See example on next slide.

More Related