Download Presentation
## CS 691 Computational Photography

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -

**CS 691 Computational Photography**Instructor: Gianfranco Doretto Cutting Images**This Lecture: Finding Seams and Boundaries**Segmentation**This Lecture: Finding Seams and Boundaries**Retargetting http://swieskowski.net/carve/**This Lecture: Finding seams and boundaries**Fundamental Concept: The Image as a Graph • Intelligent Scissors: Good boundary = short path • Graph cuts: Good region has low cutting cost**Semi-automated segmentation**User provides imprecise and incomplete specification of region – your algorithm has to read his/her mind. Key problems What groups of pixels form cohesive regions? What pixels are likely to be on the boundary of regions? Which region is the user trying to select?**What makes a good region?**• Contains small range of color/texture • Looks different than background • Compact**What makes a good boundary?**• High gradient along boundary • Gradient in right direction • Smooth**The Image as a Graph**Node: pixel Edge: cost of path or cut between two pixels**Intelligent Scissors**Mortenson and Barrett (SIGGRAPH 1995)**Intelligent Scissors**• Formulation: find good boundary between seed points • Challenges • Minimize interaction time • Define what makes a good boundary • Efficiently find it**Intelligent Scissors**Mortenson and Barrett (SIGGRAPH 1995) A good image boundary has a short path through the graph. 1 2 1 Start 2 1 9 1 5 4 4 3 3 3 6 1 1 1 End**Intelligent Scissors: method**• Define boundary cost between neighboring pixels • User specifies a starting point (seed) • Compute lowest cost from seed to each other pixel • Get new seed, get path between seeds, repeat**Intelligent Scissors: method**• Define boundary cost between neighboring pixels • Lower if edge is present (e.g., with edge(im, ‘canny’)) • Lower if gradient is strong • Lower if gradient is in direction of boundary**Gradients, Edges, and Path Cost**Gradient Magnitude Path Cost Edge Image**Intelligent Scissors: method**• Define boundary cost between neighboring pixels • User specifies a starting point (seed) • Snapping**Intelligent Scissors: method**• Define boundary cost between neighboring pixels • User specifies a starting point (seed) • Compute lowest cost from seed to each other pixel • Djikstra’s shortest path algorithm**Djikstra’s shortest path algorithm**Initialize, given seed s: • Compute cost2(q, r) % cost for boundary from pixel q to neighboring pixel r • cost(s) = 0 % total cost from seed to this point • A = {s} % set to be expanded • E = { } % set of expanded pixels • P(q) % pointer to pixel that leads to q Loop while A is not empty 1. q = pixel in A with lowest cost • for each pixel r in neighborhood of q that is not in E • cost_tmp = cost(q) + cost2(q,r) • if (r is not in A) OR (cost_tmp < cost(r)) • cost(r) = cost_tmp • P(r) = q • Add r to A**Intelligent Scissors: method**• Define boundary cost between neighboring pixels • User specifies a starting point (seed) • Compute lowest cost from seed to each other pixel • Get new seed, get path between seeds, repeat**Intelligent Scissors: improving interaction**• Snap when placing first seed • Automatically adjust as user drags • Freeze stable boundary points to make new seeds**Where will intelligent scissors work well, or have problems?****Grab cuts and graph cuts**Magic Wand(198?) Intelligent ScissorsMortensen and Barrett (1995) GrabCut User Input Result Regions Regions & Boundary Boundary Source: Rother**Segmentation with graph cuts**Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1)**Segmentation with graph cuts**Source (Label 0) Cost to assign to 0 Cost to split nodes Cost to assign to 1 Sink (Label 1)**Foreground (source)**Min Cut Background(sink) Cut:separatingsource and sink; Energy: collection of edges Min Cut: Global minimal enegry in polynomial time Interactive Graph Cuts [Boykov, Jolly ICCV’01] Image constraints**GrabCut Colour Model**Gaussian Mixture Model (typically 5-8 components) R R Iterated graph cut Foreground &Background Foreground G Background G Background Source: Rother**Graph cuts segmentation**• Define graph • usually 4-connected or 8-connected • Set weights to foreground/background • Color histogram or mixture of Gaussians for background and foreground • Set weights for edges between pixels • Apply min-cut/max-flow algorithm • Return to 2, using current labels to compute foreground, background models**What is easy or hard about these cases for graphcut-based**segmentation?**GrabCut – Interactive Foreground Extraction10**Easier examples**GrabCut – Interactive Foreground Extraction11**More difficult Examples Camouflage & Low Contrast Fine structure Harder Case Initial Rectangle InitialResult**Limitations of Graph Cuts**• Requires associative graphs • Connected nodes should prefer to have the same label • Is optimal only for binary problems**Other applications: Seam Carving**Seam Carving – Avidan and Shamir (2007) Demo: http://swieskowski.net/carve/**Other applications: Seam Carving**• Find shortest path from top to bottom (or left to right), where cost = gradient magnitude http://www.youtube.com/watch?v=6NcIJXTlugc Seam Carving – Avidan and Shamir (2007) Demo: http://swieskowski.net/carve/**Dynamic Programming**• Well known algorithm design techniques:. • Divide-and-conquer algorithms • Another strategy for designing algorithms is dynamic programming. • Used when problem breaks down into recurring small subproblems • Dynamic programming is typically applied to optimization problems. In such problem there can be many solutions. Each solution has a value, and we wish to find a solutionwith the optimal value.**Dynamic Programming**• Dynamic programming is a way of improving on inefficientdivide-and-conquer algorithms. • By “inefficient”, we mean that the same recursive call is made over and over. • If same subproblemis solved several times, we can use tableto store result of a subproblem the first time it is computed and thus never have to recompute it again. • Dynamic programming is applicable when the subproblems are dependent, that is, when subproblems share subsubproblems. • “Programming” refers to a tabular method**Elements of Dynamic Programming (DP)**DP is used to solve problems with the following characteristics: • Simple subproblems • We should be able to break the original problem to smaller subproblems that have the same structure • Optimal substructure of the problems • The optimal solutionto the problem contains within optimal solutions to its subproblems. • Overlapping sub-problems • there exist some places where we solve the same subproblemmore than once.**Steps to Designing a Dynamic Programming Algorithm**• Characterize optimal substructure 2. Recursively define the value of an optimal solution 3. Compute the value bottom up 4. (if needed) Construct an optimal solution**Example: Matrix-chain Multiplication**• Suppose we have a sequence or chain A1, A2, …, An of n matrices to be multiplied • That is, we want to compute the product A1A2…An • There are many possible ways (parenthesizations) to compute the product**Matrix-chain Multiplication …contd**• Example: consider the chain A1, A2, A3, A4 of 4 matrices • Let us compute the product A1A2A3A4 • There are 5 possible ways: • (A1(A2(A3A4))) • (A1((A2A3)A4)) • ((A1A2)(A3A4)) • ((A1(A2A3))A4) • (((A1A2)A3)A4)**Matrix-chain Multiplication …contd**• To compute the number of scalar multiplications necessary, we must know: • Algorithm to multiply two matrices • Matrix dimensions • Can you write the algorithm to multiply two matrices?**Algorithm to Multiply 2 Matrices**Input: Matrices Ap×q and Bq×r (with dimensions p×q and q×r) Result: Matrix Cp×r resulting from the product A·B MATRIX-MULTIPLY(Ap×q , Bq×r) 1. for i ← 1 top 2. for j ← 1 tor 3. C[i, j]← 0 4. for k ← 1 toq 5. C[i, j]← C[i, j] + A[i, k]· B[k, j] 6. returnC Scalar multiplication in line 5 dominates time to compute CNumber of scalar multiplications = pqr**Matrix-chain Multiplication …contd**• Example: Consider three matrices A10x100, B100x5, and C5x50 • There are 2 ways to parenthesize • ((AB)C) = D10x5·C5x50 • AB 10·100·5=5,000 scalar multiplications • DC 10·5·50 =2,500 scalar multiplications • (A(BC)) = A10100·E10050 • BC100·5·50=25,000 scalar multiplications • AE10·100·50 =50,000 scalar multiplications Total: 7,500 Total: 75,000**Matrix-chain Multiplication …contd**• Matrix-chain multiplication problem • Given a chain A1, A2, …, An of n matrices, where for i=1, 2, …, n, matrix Ai has dimension pi-1xpi • Parenthesize the product A1A2…An such that the total number of scalar multiplications is minimized • Brute force method of exhaustive search takes time exponential in n**Dynamic Programming Approach**• The structure of an optimal solution • Let us use the notation Ai..j for the matrix that results from the product Ai Ai+1 … Aj • An optimal parenthesization of the product A1A2…An splits the product between Akand Ak+1for some integer k where1 ≤ k < n • First compute matrices A1..k and Ak+1..n ; then multiply them to get the final matrix A1..n**Dynamic Programming Approach …contd**• Key observation: parenthesizations of the subchains A1A2…Ak and Ak+1Ak+2…An must also be optimal if the parenthesization of the chain A1A2…An is optimal (why?) • That is, the optimal solution to the problem contains within it the optimal solution to subproblems**Dynamic Programming Approach …contd**• Recursive definition of the value of an optimal solution • Let m[i, j] be the minimum number of scalar multiplications necessary to compute Ai..j • Minimum cost to compute A1..n is m[1, n] • Suppose the optimal parenthesization of Ai..jsplits the product between Akand Ak+1for some integer k where i ≤ k < j**Dynamic Programming Approach …contd**• Ai..j= (Ai Ai+1…Ak)·(Ak+1Ak+2…Aj)= Ai..k· Ak+1..j • Cost of computing Ai..j = cost of computing Ai..k + cost of computing Ak+1..j + cost of multiplying Ai..k and Ak+1..j • Cost of multiplying Ai..k and Ak+1..j is pi-1pk pj • m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj for i ≤ k < j • m[i, i ] = 0 for i=1,2,…,n**Dynamic Programming Approach …contd**• But… optimal parenthesization occurs at one value of k among all possible i ≤ k < j • Check all these and select the best one 0 if i=j m[i, j ] = min {m[i, k] + m[k+1, j ] + pi-1pk pj}if i<j i≤ k< j