Cs 691 computational photography
This presentation is the property of its rightful owner.
Sponsored Links
1 / 59

CS 691 Computational Photography PowerPoint PPT Presentation


  • 94 Views
  • Uploaded on
  • Presentation posted in: General

CS 691 Computational Photography. Instructor: Gianfranco Doretto Cutting Images. This Lecture: Finding Seams and Boundaries. Segmentation. This Lecture: Finding Seams and Boundaries. Retargetting. http://swieskowski.net/carve/. This Lecture: Finding Seams and Boundaries. Stitching.

Download Presentation

CS 691 Computational Photography

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Cs 691 computational photography

CS 691 Computational Photography

Instructor: Gianfranco Doretto

Cutting Images


This lecture finding seams and boundaries

This Lecture: Finding Seams and Boundaries

Segmentation


This lecture finding seams and boundaries1

This Lecture: Finding Seams and Boundaries

Retargetting

http://swieskowski.net/carve/


This lecture finding seams and boundaries2

This Lecture: Finding Seams and Boundaries

Stitching


This lecture finding seams and boundaries3

This Lecture: Finding seams and boundaries

Fundamental Concept: The Image as a Graph

  • Intelligent Scissors: Good boundary = short path

  • Graph cuts: Good region has low cutting cost


Semi automated segmentation

Semi-automated segmentation

User provides imprecise and incomplete specification of region – your algorithm has to read his/her mind.

Key problems

What groups of pixels form cohesive regions?

What pixels are likely to be on the boundary of regions?

Which region is the user trying to select?


What makes a good region

What makes a good region?

  • Contains small range of color/texture

  • Looks different than background

  • Compact


What makes a good boundary

What makes a good boundary?

  • High gradient along boundary

  • Gradient in right direction

  • Smooth


The image as a graph

The Image as a Graph

Node: pixel

Edge: cost of path or cut between two pixels


Intelligent scissors

Intelligent Scissors

Mortenson and Barrett (SIGGRAPH 1995)


Intelligent scissors1

Intelligent Scissors

  • Formulation: find good boundary between seed points

  • Challenges

    • Minimize interaction time

    • Define what makes a good boundary

    • Efficiently find it


Intelligent scissors2

Intelligent Scissors

Mortenson and Barrett (SIGGRAPH 1995)

A good image boundary has a short path through the graph.

1

2

1

Start

2

1

9

1

5

4

4

3

3

3

6

1

1

1

End


Intelligent scissors method

Intelligent Scissors: method

  • Define boundary cost between neighboring pixels

  • User specifies a starting point (seed)

  • Compute lowest cost from seed to each other pixel

  • Get new seed, get path between seeds, repeat


Intelligent scissors method1

Intelligent Scissors: method

  • Define boundary cost between neighboring pixels

    • Lower if edge is present (e.g., with edge(im, ‘canny’))

    • Lower if gradient is strong

    • Lower if gradient is in direction of boundary


Gradients edges and path cost

Gradients, Edges, and Path Cost

Gradient Magnitude

Path Cost

Edge Image


Intelligent scissors method2

Intelligent Scissors: method

  • Define boundary cost between neighboring pixels

  • User specifies a starting point (seed)

    • Snapping


Intelligent scissors method3

Intelligent Scissors: method

  • Define boundary cost between neighboring pixels

  • User specifies a starting point (seed)

  • Compute lowest cost from seed to each other pixel

    • Djikstra’s shortest path algorithm


Djikstra s shortest path algorithm

Djikstra’s shortest path algorithm

Initialize, given seed s:

  • Compute cost2(q, r) % cost for boundary from pixel q to neighboring pixel r

  • cost(s) = 0 % total cost from seed to this point

  • A = {s} % set to be expanded

  • E = { } % set of expanded pixels

  • P(q) % pointer to pixel that leads to q

    Loop while A is not empty

    1. q = pixel in A with lowest cost

  • for each pixel r in neighborhood of q that is not in E

    • cost_tmp = cost(q) + cost2(q,r)

    • if (r is not in A) OR (cost_tmp < cost(r))

      • cost(r) = cost_tmp

      • P(r) = q

      • Add r to A


Intelligent scissors method4

Intelligent Scissors: method

  • Define boundary cost between neighboring pixels

  • User specifies a starting point (seed)

  • Compute lowest cost from seed to each other pixel

  • Get new seed, get path between seeds, repeat


Intelligent scissors improving interaction

Intelligent Scissors: improving interaction

  • Snap when placing first seed

  • Automatically adjust as user drags

  • Freeze stable boundary points to make new seeds


Where will intelligent scissors work well or have problems

Where will intelligent scissors work well, or have problems?


Grab cuts and graph cuts

Grab cuts and graph cuts

Magic Wand(198?)

Intelligent ScissorsMortensen and Barrett (1995)

GrabCut

User Input

Result

Regions

Regions & Boundary

Boundary

Source: Rother


Segmentation with graph cuts

Segmentation with graph cuts

Source (Label 0)

Cost to assign to 0

Cost to split nodes

Cost to assign to 1

Sink (Label 1)


Segmentation with graph cuts1

Segmentation with graph cuts

Source (Label 0)

Cost to assign to 0

Cost to split nodes

Cost to assign to 1

Sink (Label 1)


Interactive graph cuts boykov jolly iccv 01

Foreground (source)

Min Cut

Background(sink)

Cut:separatingsource and sink; Energy: collection of edges

Min Cut: Global minimal enegry in polynomial time

Interactive Graph Cuts [Boykov, Jolly ICCV’01]

Image

constraints


Grabcut colour model

GrabCut Colour Model

Gaussian Mixture Model (typically 5-8 components)

R

R

Iterated graph cut

Foreground &Background

Foreground

G

Background

G

Background

Source: Rother


Graph cuts segmentation

Graph cuts segmentation

  • Define graph

    • usually 4-connected or 8-connected

  • Set weights to foreground/background

    • Color histogram or mixture of Gaussians for background and foreground

  • Set weights for edges between pixels

  • Apply min-cut/max-flow algorithm

  • Return to 2, using current labels to compute foreground, background models


What is easy or hard about these cases for graphcut based segmentation

What is easy or hard about these cases for graphcut-based segmentation?


Easier examples

GrabCut – Interactive Foreground Extraction10

Easier examples


More difficult examples

GrabCut – Interactive Foreground Extraction11

More difficult Examples

Camouflage &

Low Contrast

Fine structure

Harder Case

Initial Rectangle

InitialResult


Lazy snapping li et al sg 2004

Lazy Snapping (Li et al. SG 2004)


Limitations of graph cuts

Limitations of Graph Cuts

  • Requires associative graphs

    • Connected nodes should prefer to have the same label

  • Is optimal only for binary problems


Other applications seam carving

Other applications: Seam Carving

Seam Carving – Avidan and Shamir (2007)

Demo: http://swieskowski.net/carve/


Other applications seam carving1

Other applications: Seam Carving

  • Find shortest path from top to bottom (or left to right), where cost = gradient magnitude

http://www.youtube.com/watch?v=6NcIJXTlugc

Seam Carving – Avidan and Shamir (2007)

Demo: http://swieskowski.net/carve/


Dynamic programming

Dynamic Programming

  • Well known algorithm design techniques:.

    • Divide-and-conquer algorithms

  • Another strategy for designing algorithms is dynamic programming.

    • Used when problem breaks down into recurring small subproblems

  • Dynamic programming is typically applied to optimization problems. In such problem there can be many solutions. Each solution has a value, and we wish to find a solutionwith the optimal value.


Dynamic programming1

Dynamic Programming

  • Dynamic programming is a way of improving on inefficientdivide-and-conquer algorithms.

  • By “inefficient”, we mean that the same recursive call is made over and over.

  • If same subproblemis solved several times, we can use tableto store result of a subproblem the first time it is computed and thus never have to recompute it again.

  • Dynamic programming is applicable when the subproblems are dependent, that is, when subproblems share subsubproblems.

  • “Programming” refers to a tabular method


Elements of dynamic programming dp

Elements of Dynamic Programming (DP)

DP is used to solve problems with the following characteristics:

  • Simple subproblems

    • We should be able to break the original problem to smaller subproblems that have the same structure

  • Optimal substructure of the problems

    • The optimal solutionto the problem contains within optimal solutions to its subproblems.

  • Overlapping sub-problems

    • there exist some places where we solve the same subproblemmore than once.


Steps to designing a dynamic programming algorithm

Steps to Designing a Dynamic Programming Algorithm

  • Characterize optimal substructure

    2. Recursively define the value of an optimal solution

    3. Compute the value bottom up

    4. (if needed) Construct an optimal solution


Example matrix chain multiplication

Example: Matrix-chain Multiplication

  • Suppose we have a sequence or chain A1, A2, …, An of n matrices to be multiplied

    • That is, we want to compute the product A1A2…An

  • There are many possible ways (parenthesizations) to compute the product


Matrix chain multiplication contd

Matrix-chain Multiplication …contd

  • Example: consider the chain A1, A2, A3, A4 of 4 matrices

    • Let us compute the product A1A2A3A4

  • There are 5 possible ways:

    • (A1(A2(A3A4)))

    • (A1((A2A3)A4))

    • ((A1A2)(A3A4))

    • ((A1(A2A3))A4)

    • (((A1A2)A3)A4)


Matrix chain multiplication contd1

Matrix-chain Multiplication …contd

  • To compute the number of scalar multiplications necessary, we must know:

    • Algorithm to multiply two matrices

    • Matrix dimensions

  • Can you write the algorithm to multiply two matrices?


Algorithm to multiply 2 matrices

Algorithm to Multiply 2 Matrices

Input: Matrices Ap×q and Bq×r (with dimensions p×q and q×r)

Result: Matrix Cp×r resulting from the product A·B

MATRIX-MULTIPLY(Ap×q , Bq×r)

1.for i ← 1 top

2.for j ← 1 tor

3.C[i, j]← 0

4.for k ← 1 toq

5.C[i, j]← C[i, j] + A[i, k]· B[k, j]

6.returnC

Scalar multiplication in line 5 dominates time to compute CNumber of scalar multiplications = pqr


Matrix chain multiplication contd2

Matrix-chain Multiplication …contd

  • Example: Consider three matrices A10x100, B100x5, and C5x50

  • There are 2 ways to parenthesize

    • ((AB)C) = D10x5·C5x50

      • AB 10·100·5=5,000 scalar multiplications

      • DC 10·5·50 =2,500 scalar multiplications

    • (A(BC)) = A10100·E10050

      • BC100·5·50=25,000 scalar multiplications

      • AE10·100·50 =50,000 scalar multiplications

Total: 7,500

Total: 75,000


Matrix chain multiplication contd3

Matrix-chain Multiplication …contd

  • Matrix-chain multiplication problem

    • Given a chain A1, A2, …, An of n matrices, where for i=1, 2, …, n, matrix Ai has dimension pi-1xpi

    • Parenthesize the product A1A2…An such that the total number of scalar multiplications is minimized

  • Brute force method of exhaustive search takes time exponential in n


Dynamic programming approach

Dynamic Programming Approach

  • The structure of an optimal solution

    • Let us use the notation Ai..j for the matrix that results from the product Ai Ai+1 … Aj

    • An optimal parenthesization of the product A1A2…An splits the product between Akand Ak+1for some integer k where1 ≤ k < n

    • First compute matrices A1..k and Ak+1..n ; then multiply them to get the final matrix A1..n


Dynamic programming approach contd

Dynamic Programming Approach …contd

  • Key observation: parenthesizations of the subchains A1A2…Ak and Ak+1Ak+2…An must also be optimal if the parenthesization of the chain A1A2…An is optimal (why?)

  • That is, the optimal solution to the problem contains within it the optimal solution to subproblems


Dynamic programming approach contd1

Dynamic Programming Approach …contd

  • Recursive definition of the value of an optimal solution

    • Let m[i, j] be the minimum number of scalar multiplications necessary to compute Ai..j

    • Minimum cost to compute A1..n is m[1, n]

    • Suppose the optimal parenthesization of Ai..jsplits the product between Akand Ak+1for some integer k where i ≤ k < j


Dynamic programming approach contd2

Dynamic Programming Approach …contd

  • Ai..j= (Ai Ai+1…Ak)·(Ak+1Ak+2…Aj)= Ai..k· Ak+1..j

  • Cost of computing Ai..j = cost of computing Ai..k + cost of computing Ak+1..j + cost of multiplying Ai..k and Ak+1..j

  • Cost of multiplying Ai..k and Ak+1..j is pi-1pk pj

  • m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj for i ≤ k < j

  • m[i, i ] = 0 for i=1,2,…,n


Dynamic programming approach contd3

Dynamic Programming Approach …contd

  • But… optimal parenthesization occurs at one value of k among all possible i ≤ k < j

  • Check all these and select the best one

0 if i=j

m[i, j ] =

min {m[i, k] + m[k+1, j ] + pi-1pk pj}if i<j

i≤ k< j


Dynamic programming approach contd4

Dynamic Programming Approach …contd

  • To keep track of how to construct an optimal solution, we use a table s

  • s[i, j ] = value of k at which Ai Ai+1 … Ajis split for optimal parenthesization

  • Algorithm: next slide

    • First computes costs for chains of length l=1

    • Then for chains of length l=2,3, … and so on

    • Computes the optimal cost bottom-up


Algorithm to compute optimal cost

Algorithm to Compute Optimal Cost

Input: Array p[0…n] containing matrix dimensions and n

Result: Minimum-cost table m and split table s

MATRIX-CHAIN-ORDER(p[ ], n)

for i← 1 ton

m[i, i]← 0

for l← 2 ton

for i← 1 ton-l+1

j← i+l-1

m[i, j]← ∞

for k← itoj-1

q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]

ifq < m[i, j]

m[i, j]← q

s[i, j]← k

returnm and s

Takes O(n3) time

Requires O(n2) space


Constructing optimal solution

Constructing Optimal Solution

  • Our algorithm computes the minimum-cost table m and the split table s

  • The optimal solution can be constructed from the split table s

    • Each entry s[i, j ]=k shows where to split the product Ai Ai+1 … Ajfor the minimum cost


Example

Example

  • Show how to multiply this matrix chain optimally

  • Solution on the board

    • Minimum cost 15,125

    • Optimal parenthesization ((A1(A2A3))((A4 A5)A6))


Minimal error boundary

2

_

=

overlap error

min. error boundary

Minimal error boundary

overlapping blocks

vertical boundary


Other applications stitching

Other applications: stitching

Graphcut Textures – Kwatra et al. SIGGRAPH 2003


Other applications stitching1

Other applications: stitching

Graphcut Textures – Kwatra et al. SIGGRAPH 2003

+

Ideal boundary:

Similar color in both images

High gradient in both images


Summary of big ideas

Summary of big ideas

  • Treat image as a graph

    • Pixels are nodes

    • Between-pixel edge weights based on gradients

    • Sometimes per-pixel weights for affinity to foreground/background

  • Good boundaries are a short path through the graph (Intelligent Scissors, Seam Carving)

  • Good regions are produced by a low-cost cut (GrabCuts, Graph Cut Stitching)


Slide credits

Slide Credits

  • This set of sides also contains contributions kindly made available by the following authors

    • Alexei Efros

    • CarstenRother

    • Derek Hoiem


  • Login