- 107 Views
- Uploaded on
- Presentation posted in: General

CS 691 Computational Photography

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

CS 691 Computational Photography

Instructor: Gianfranco Doretto

Cutting Images

Segmentation

Retargetting

http://swieskowski.net/carve/

Stitching

Fundamental Concept: The Image as a Graph

- Intelligent Scissors: Good boundary = short path
- Graph cuts: Good region has low cutting cost

User provides imprecise and incomplete specification of region – your algorithm has to read his/her mind.

Key problems

What groups of pixels form cohesive regions?

What pixels are likely to be on the boundary of regions?

Which region is the user trying to select?

- Contains small range of color/texture
- Looks different than background
- Compact

- High gradient along boundary
- Gradient in right direction
- Smooth

Node: pixel

Edge: cost of path or cut between two pixels

Mortenson and Barrett (SIGGRAPH 1995)

- Formulation: find good boundary between seed points
- Challenges
- Minimize interaction time
- Define what makes a good boundary
- Efficiently find it

Mortenson and Barrett (SIGGRAPH 1995)

A good image boundary has a short path through the graph.

1

2

1

Start

2

1

9

1

5

4

4

3

3

3

6

1

1

1

End

- Define boundary cost between neighboring pixels
- User specifies a starting point (seed)
- Compute lowest cost from seed to each other pixel
- Get new seed, get path between seeds, repeat

- Define boundary cost between neighboring pixels
- Lower if edge is present (e.g., with edge(im, ‘canny’))
- Lower if gradient is strong
- Lower if gradient is in direction of boundary

Gradient Magnitude

Path Cost

Edge Image

- Define boundary cost between neighboring pixels
- User specifies a starting point (seed)
- Snapping

- Define boundary cost between neighboring pixels
- User specifies a starting point (seed)
- Compute lowest cost from seed to each other pixel
- Djikstra’s shortest path algorithm

Initialize, given seed s:

- Compute cost2(q, r) % cost for boundary from pixel q to neighboring pixel r
- cost(s) = 0 % total cost from seed to this point
- A = {s} % set to be expanded
- E = { } % set of expanded pixels
- P(q) % pointer to pixel that leads to q
Loop while A is not empty

1. q = pixel in A with lowest cost

- for each pixel r in neighborhood of q that is not in E
- cost_tmp = cost(q) + cost2(q,r)
- if (r is not in A) OR (cost_tmp < cost(r))
- cost(r) = cost_tmp
- P(r) = q
- Add r to A

- Define boundary cost between neighboring pixels
- User specifies a starting point (seed)
- Compute lowest cost from seed to each other pixel
- Get new seed, get path between seeds, repeat

- Snap when placing first seed
- Automatically adjust as user drags
- Freeze stable boundary points to make new seeds

Magic Wand(198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

User Input

Result

Regions

Regions & Boundary

Boundary

Source: Rother

Source (Label 0)

Cost to assign to 0

Cost to split nodes

Cost to assign to 1

Sink (Label 1)

Source (Label 0)

Cost to assign to 0

Cost to split nodes

Cost to assign to 1

Sink (Label 1)

Foreground (source)

Min Cut

Background(sink)

Cut:separatingsource and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Image

constraints

Gaussian Mixture Model (typically 5-8 components)

R

R

Iterated graph cut

Foreground &Background

Foreground

G

Background

G

Background

Source: Rother

- Define graph
- usually 4-connected or 8-connected

- Set weights to foreground/background
- Color histogram or mixture of Gaussians for background and foreground

- Set weights for edges between pixels
- Apply min-cut/max-flow algorithm
- Return to 2, using current labels to compute foreground, background models

GrabCut – Interactive Foreground Extraction10

GrabCut – Interactive Foreground Extraction11

Camouflage &

Low Contrast

Fine structure

Harder Case

Initial Rectangle

InitialResult

- Requires associative graphs
- Connected nodes should prefer to have the same label

- Is optimal only for binary problems

Seam Carving – Avidan and Shamir (2007)

Demo: http://swieskowski.net/carve/

- Find shortest path from top to bottom (or left to right), where cost = gradient magnitude

http://www.youtube.com/watch?v=6NcIJXTlugc

Seam Carving – Avidan and Shamir (2007)

Demo: http://swieskowski.net/carve/

- Well known algorithm design techniques:.
- Divide-and-conquer algorithms

- Another strategy for designing algorithms is dynamic programming.
- Used when problem breaks down into recurring small subproblems

- Dynamic programming is typically applied to optimization problems. In such problem there can be many solutions. Each solution has a value, and we wish to find a solutionwith the optimal value.

- Dynamic programming is a way of improving on inefficientdivide-and-conquer algorithms.
- By “inefficient”, we mean that the same recursive call is made over and over.
- If same subproblemis solved several times, we can use tableto store result of a subproblem the first time it is computed and thus never have to recompute it again.
- Dynamic programming is applicable when the subproblems are dependent, that is, when subproblems share subsubproblems.
- “Programming” refers to a tabular method

DP is used to solve problems with the following characteristics:

- Simple subproblems
- We should be able to break the original problem to smaller subproblems that have the same structure

- Optimal substructure of the problems
- The optimal solutionto the problem contains within optimal solutions to its subproblems.

- Overlapping sub-problems
- there exist some places where we solve the same subproblemmore than once.

- Characterize optimal substructure
2. Recursively define the value of an optimal solution

3. Compute the value bottom up

4. (if needed) Construct an optimal solution

- Suppose we have a sequence or chain A1, A2, …, An of n matrices to be multiplied
- That is, we want to compute the product A1A2…An

- There are many possible ways (parenthesizations) to compute the product

- Example: consider the chain A1, A2, A3, A4 of 4 matrices
- Let us compute the product A1A2A3A4

- There are 5 possible ways:
- (A1(A2(A3A4)))
- (A1((A2A3)A4))
- ((A1A2)(A3A4))
- ((A1(A2A3))A4)
- (((A1A2)A3)A4)

- To compute the number of scalar multiplications necessary, we must know:
- Algorithm to multiply two matrices
- Matrix dimensions

- Can you write the algorithm to multiply two matrices?

Input: Matrices Ap×q and Bq×r (with dimensions p×q and q×r)

Result: Matrix Cp×r resulting from the product A·B

MATRIX-MULTIPLY(Ap×q , Bq×r)

1.for i ← 1 top

2.for j ← 1 tor

3.C[i, j]← 0

4.for k ← 1 toq

5.C[i, j]← C[i, j] + A[i, k]· B[k, j]

6.returnC

Scalar multiplication in line 5 dominates time to compute CNumber of scalar multiplications = pqr

- Example: Consider three matrices A10x100, B100x5, and C5x50
- There are 2 ways to parenthesize
- ((AB)C) = D10x5·C5x50
- AB 10·100·5=5,000 scalar multiplications
- DC 10·5·50 =2,500 scalar multiplications

- (A(BC)) = A10100·E10050
- BC100·5·50=25,000 scalar multiplications
- AE10·100·50 =50,000 scalar multiplications

- ((AB)C) = D10x5·C5x50

Total: 7,500

Total: 75,000

- Matrix-chain multiplication problem
- Given a chain A1, A2, …, An of n matrices, where for i=1, 2, …, n, matrix Ai has dimension pi-1xpi
- Parenthesize the product A1A2…An such that the total number of scalar multiplications is minimized

- Brute force method of exhaustive search takes time exponential in n

- The structure of an optimal solution
- Let us use the notation Ai..j for the matrix that results from the product Ai Ai+1 … Aj
- An optimal parenthesization of the product A1A2…An splits the product between Akand Ak+1for some integer k where1 ≤ k < n
- First compute matrices A1..k and Ak+1..n ; then multiply them to get the final matrix A1..n

- Key observation: parenthesizations of the subchains A1A2…Ak and Ak+1Ak+2…An must also be optimal if the parenthesization of the chain A1A2…An is optimal (why?)
- That is, the optimal solution to the problem contains within it the optimal solution to subproblems

- Recursive definition of the value of an optimal solution
- Let m[i, j] be the minimum number of scalar multiplications necessary to compute Ai..j
- Minimum cost to compute A1..n is m[1, n]
- Suppose the optimal parenthesization of Ai..jsplits the product between Akand Ak+1for some integer k where i ≤ k < j

- Ai..j= (Ai Ai+1…Ak)·(Ak+1Ak+2…Aj)= Ai..k· Ak+1..j
- Cost of computing Ai..j = cost of computing Ai..k + cost of computing Ak+1..j + cost of multiplying Ai..k and Ak+1..j
- Cost of multiplying Ai..k and Ak+1..j is pi-1pk pj
- m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj for i ≤ k < j
- m[i, i ] = 0 for i=1,2,…,n

- But… optimal parenthesization occurs at one value of k among all possible i ≤ k < j
- Check all these and select the best one

0 if i=j

m[i, j ] =

min {m[i, k] + m[k+1, j ] + pi-1pk pj}if i<j

i≤ k< j

- To keep track of how to construct an optimal solution, we use a table s
- s[i, j ] = value of k at which Ai Ai+1 … Ajis split for optimal parenthesization
- Algorithm: next slide
- First computes costs for chains of length l=1
- Then for chains of length l=2,3, … and so on
- Computes the optimal cost bottom-up

Input: Array p[0…n] containing matrix dimensions and n

Result: Minimum-cost table m and split table s

MATRIX-CHAIN-ORDER(p[ ], n)

for i← 1 ton

m[i, i]← 0

for l← 2 ton

for i← 1 ton-l+1

j← i+l-1

m[i, j]← ∞

for k← itoj-1

q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]

ifq < m[i, j]

m[i, j]← q

s[i, j]← k

returnm and s

Takes O(n3) time

Requires O(n2) space

- Our algorithm computes the minimum-cost table m and the split table s
- The optimal solution can be constructed from the split table s
- Each entry s[i, j ]=k shows where to split the product Ai Ai+1 … Ajfor the minimum cost

- Show how to multiply this matrix chain optimally
- Solution on the board
- Minimum cost 15,125
- Optimal parenthesization ((A1(A2A3))((A4 A5)A6))

2

_

=

overlap error

min. error boundary

overlapping blocks

vertical boundary

Graphcut Textures – Kwatra et al. SIGGRAPH 2003

Graphcut Textures – Kwatra et al. SIGGRAPH 2003

+

Ideal boundary:

Similar color in both images

High gradient in both images

- Treat image as a graph
- Pixels are nodes
- Between-pixel edge weights based on gradients
- Sometimes per-pixel weights for affinity to foreground/background

- Good boundaries are a short path through the graph (Intelligent Scissors, Seam Carving)
- Good regions are produced by a low-cost cut (GrabCuts, Graph Cut Stitching)

- This set of sides also contains contributions kindly made available by the following authors
- Alexei Efros
- CarstenRother
- Derek Hoiem