Matting
Download
1 / 86

Matting - PowerPoint PPT Presentation


  • 157 Views
  • Uploaded on

Matting. Roey Izkovsky Yuval Kaminka. Helping Superman fly since 1978. Outline. The matting problem Previous work New approaches: The iterative approach Jue Wang, Michael F.Cohen Closed form solution Anat Levin, Dani Lischinski,Yair Weiss Comparison and summary Bonus?. Outline.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Matting' - fairfax


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Matting

Matting

Roey Izkovsky

Yuval Kaminka

Helping Superman fly since 1978


Outline
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


Outline1
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


The matting problem motivation
The matting problem - Motivation

 Image and video editing

New background

Composite image


The matting problem motivation1
The matting problem - Motivation

 Image and video editing

Input image

New image


The matting problem
The matting problem

  • The separation of an image I into

    • Foreground object image F

    • Background image B

    • Alpha matte α – the opacity

  • Problem: extract F, B, α from image

hair

fur


Why is matting challenging
Why is matting challenging?

  • Under constrained problem:One equation, 3 unknowns

 We need to constrain the problem!


Outline2
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


Previous work
Previous work

Two types:

Known background Natural image

matting matting


Known background
Known Background

  • Blue screen Matting

  • Still under-constrained

    • Solution: make more assumptions

      • “Foreground contains no blue”

      • Other foreground distribution assumption…

  • Use two different backgrounds

  • Main flaw: need to know the background…

Blue background

Composite image


Natural image matting
Natural Image Matting

  • The assumptions:

    • Smoothness of the alpha matte

    • GMM for the Background and Foreground colors

  • Initial estimate:trimap provided by the user

Background

Foreground

Unknown

Input image

Trimap


Natural image matting1
Natural Image Matting

  • The algorithms framework:

    • Estimate F, B distributions from close pixels

    • Find best α by some method


Knockout
Knockout

  • Extrapolate F,B from close neighborhood

  • Estimate α from calculated F, B values


Bayesian
Bayesian

  • Estimate F, B distributions in area

  • Find best α matching distributions


Bayesian1
Bayesian

  • P(F), P(B) from image samples

  • P(C|F,B,α) using a distribution for C


Natural image matting2
Natural Image Matting

  • Main flaw: Accurate trimap required

    • Tedious to provide manually

  • Hard to extract automatically

 In particular, not feasible to videos

Input image

Trimap

Binary segmentation

Adding unknown region


Great.So let’s get started…


Outline3
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


New approach to matting
New Approach to Matting

Trimap reduces to scribbles

Two new methods

  • Iterative optimization approach

    • Heuristic algorithmic optimization

  • A closed form solution

    • Mathematical approach

Trimap

Scribbles


Iterative optimization approach

Iterative optimization approach

Jue Wang

Michael F. Cohen



Iterative approach1
Iterative approach

  • Score:

    fit to image data +alpha matte smoothness

  • Iteratively propagating estimated results.


Iterative optimization outline
Iterative optimization - outline

  • Initialize “work pixels” from scribbles

  • Repeatedly:

    • Expand work pixels

    • Find best alpha matte

  • Stop when finished


Initialization

Uc = {user scribbles}

ui = 0

α = 0

ui = 0

α = 1

ui = 1

α = 0.5

Initialization

  • Introducing:

    • ui - uncertainty variable

    • Uc – work pixels


Optimization
Optimization

Uc = {user scribbles + 15 pixel radius}

Our goal:

find α matte for Uc that minimizes the energy -

Smoothness

Data


N Possible values for F

N Possible values for B

Vd

Score for αp = α

Image color Ip


Vd

  • Fit measure of αp to Ip

  • Score for αp = α :

Fi , Bj – possible values for F, B in the pixel

wFi, wBj – corresponding weights


α = 0.4

u = 0.5

α = 0.8

u = 0.3

α = 0.4

u = 0.4

α = 0.2

u = 0.3

α = 0.9

u = 0.2

α = 0.3

u = 0.3

Vd

Fi , Bj – possible values for F, B in the pixel

wFi, wBj – corresponding weights

F Samples

B Samples

What happens when there are not enough F/B samples?

p

α = 0.5

u = 1.0


Vd

  • Score for αp = α :

  • Discretize

  • and normalize


Vs

  • Matte smoothness :


Iterative optimization step 2
Iterative optimization – step 2

Uc = {user scribbles + 15 pixel radius}

Our goal: find α matte for Uc that minimizes the energy -

Uc Graph

Nodes = Pixels, Edges by 4-connectivity


Iterative optimization step 21
Iterative optimization – step 2

GOAL: Minimize

BELIEF PROPAGATION


Iterative optimization step 22
Iterative optimization – step 2

GOAL: Minimize

BELIEF PROPAGATION

t=0

y

mpq – message from p to q

q

p

Vector: p’s “opinion” for each

possible α for q


Iterative optimization step 23
Iterative optimization – step 2

GOAL: Minimize

BELIEF PROPAGATION

t=1

y

mpq – new message pq

myp – previous message yp

q

p


Iterative optimization step 24
Iterative optimization – step 2

GOAL: Minimize

BELIEF PROPAGATION

t=2,3,4…

y

q

p


Iterative optimization step 25
Iterative optimization – step 2

GOAL: Minimize

BELIEF PROPAGATION

t=T (stopping time)

y

q

p


Iterative optimization step 26
Iterative optimization – step 2

GOAL: Minimize

BELIEF PROPAGATION

t=T (stopping time)

y

q

p

Best state calculated for each node:


Iterative optimization step 3
Iterative optimization – step 3

Found α matte for Uc that minimizes the energy -

Update F, B and uncertainty:


Iterative optimization algorithm
Iterative optimization - algorithm

  • Initialize Uc, F, B, u and alpha matte from scribbles

  • Repeatedly:

    • Expand Uc by another 15 pixel radius

    • Find best alpha matte (BP)

    • Update F,B,u for new matte

  • Stop when total uncertainty is minimal

Initial matte

Propagation of α matte

Final matte


Iterative optimization results
Iterative optimization - Results

Input image

Extracted matte


Iterative optimization results1
Iterative optimization - Results

Input image

Composite image

Extracted matte



Iterative optimization results3
Iterative optimization - Results

Scribbles result

Trimap result

Ambiguity bunny with trimap

Ambiguity bunny with scribbles


Iterative optimization summary
Iterative optimization - Summary

  • Minimal user input

  • Applicable to video

  • Sensitive to ambiguity in F, B

  • Uses simple color-model

  • Performance:

    • 15-20 min. on a 640x480 image

    • Factor 50 reported by better implementation


Fantastic.Let’s go on…


Outline4
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


Closed form solution

Closed form solution

Anat Levin

Dani Lischinski

Yair Weiss


Closed form solution1
Closed form solution

  • Assumption: local smoothness in F, B

     cancel out unknowns from the matte equs.

  • Solve for F,B and alphausing algebraic tricks.


Closed form solution2
Closed form solution

Assumptions:

  • F,B locally smooth.

     treat F,B as constant in a small window w


Closed form solution3
Closed form solution

GOAL:

Minimize -

  • Numerical stability

  • Bias to smoother matte

wj


Closed form solution4
Closed form solution

  • GOAL:

    • Minimize:


Closed form solution5
Closed form solution

  • Minimize:

3N Variables (N = image size) We can rid a, b by algebraic manipulation


Closed form solution6
Closed form solution

  • Minimize:

Theorem: for we have

Intuitively, L is some covariance matrix


Closed form solution7
Closed form solution

  • Minimize:

Proof: Rewrite in matrix form:


Closed form solution8
Closed form solution

  • Minimize:

Proof: Rewrite in matrix form:

By mean-least-squares, best a,b pair

for each window is:


Closed form solution9
Closed form solution

  • Some more manipulation give the required result

EXCITED?

GET YOUR I LOVE MATH

T-SHIRT, NOW FOR ONLY $1999


Closed form solution10

R

F2

F1

G

Closed form solution

  • For color images:

    • Simple: Do each channel separately

    • Smart: Assume one alpha for R,G,B.Use redundancy to allow a “color-line” model per window

Color line model:

OUT: F, B Constant within a window

IN: F, B are on some line


Closed form solution11
Closed form solution

  • For color images:

    • Simple: Do each channel separately

    • Smart: Assume one alpha for R,G,B.Use redundancy to allow a “color-line” model per window


Closed form solution12
Closed form solution

  • For color images:

    • Simple: Do each channel separately

    • Smart: Assume one alpha for R,G,B.Use redundancy to allow a “color-line” model per window

Now, as before, cost is:

And a,b can be cancelled out.


Closed form solution13
Closed form solution

Now problem reduced to finding best α for:

L is Huge size NxN (N = # image pixels)

But Sparse…


Closed form solution14
Closed form solution

  • The algorithm:

    • Compute L

    • Solve for given the scribbles.

      • Solving a sparse set of bilinear equationswith constraints (Lagrange multipliers)

    • Find F, B given the matte

      • Adding smoothness assumptions on F, B

  • Improvements:

    • Use larger environment in low cost by “pyramids”


Closed form solution results
Closed form solution - Results

Extracted matte

Input image


Closed form solution results1
Closed form solution - Results

Input image with scribbles

Problematic matte


Eigenvectors as guides
Eigenvectors as guides

Small eigenvectors of L are

correlated with minimal matte

L is positive definite.

Eigenbasis: v1,…,vN

Eigenvalues: λ1 > λ2 > … > λN > 0


Eigenvectors as guides1
Eigenvectors as guides

Small eigenvectors of L are

correlated with minimal matte


Eigenvectors as guides2
Eigenvectors as guides

Small eigenvectors of L are

correlated with minimal matte

 can guide user scribbles

Eigenvectors matching smallest eigenvalues

Resulting matte

Guided scribbles


Closed form solution summary
Closed form solution - Summary

  • Minimal user input

  • Provable optimality (under assumptions)

  • Assumes only smooth F,B (no color model)

  • Applicable to video (as we speak…)

  • Problematic with textures

  • Performance:

    • 20-40 seconds for a 200x300 image

    • Expensive in memory


Superb.Let’s sum up…


Outline5
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


Comparison
Comparison

Input image

Matte ground truth

Closed form solution

Iterative approach

Poisson



Comparison1
Comparison

Color ambiguity

Iterative approach Closed form

Sensitive Sensitive

Solvable by adding more scribbles


Comparison2
Comparison

Improving results…

Ambiguity bunny

Iterative approach

Bayesian

Closed form solution


Comparison3
Comparison

Optimality?

Iterative approach Closed form

Provably optimal

But for the specific

(simplified) cost

Uses heuristics

to optimize


Comparison4
Comparison

Textures

Iterative approach Closed form

F,B must satisfy

color-line model

Assumes only

Alpha matte smooth


Comparison5
Comparison

Rough edges

Iterative approach Closed form

Input image with scribbles

Can handle rough

edges

Assumes

Alpha matte smooth

 matte results 


Comparison6
Comparison

Running time

Iterative approach Closed form

20/40 seconds

Costly in memory

~20 sec.

(For medium size image)


Comparison7
Comparison

Tests

Iterative approach Closed form

Extensively tested

quantitative results

No quantitative

results reported


Outline6
Outline

  • The matting problem

  • Previous work

  • New approaches:

    • The iterative approach Jue Wang,Michael F.Cohen

    • Closed form solutionAnat Levin, Dani Lischinski,Yair Weiss

  • Comparison and summary

  • Bonus?


Environment Matting and Compositing

Douglas E. Zongker ~ Dawn M. Werner ~ Brian Curless ~ David H. Salsin


Environment matting
Environment Matting

C = F + (1- a)B + F

  • F ~ Contribution of light from Environment that travels through the object

R – reflectance image

T – Texture image


Environment matting1
Environment Matting?

Alpha Matte Environment Matte Photograph


Environment mattin
Environment Mattin

Alpha Matte Environment Matte Photograph


Summary
Summary

  • The matting problem

  • Old methods: require trimap

  • Two new methods from scribbles:

    • Iterative optimization

      • Assume: matte smooth, F,B locally similar

      • Use heuristic optimization for alpha

    • Close form solution

      • Assume: F, B locally smooth (color-line model)

      • Solve linear equations for alpha



ad