Face transfer with multilinear models
This presentation is the property of its rightful owner.
Sponsored Links
1 / 25

Face Transfer with Multilinear Models PowerPoint PPT Presentation


  • 89 Views
  • Uploaded on
  • Presentation posted in: General

Face Transfer with Multilinear Models. Daniel Vlasic & Jovan Popovic CSAIL MIT Matthew Brand & Hanspeter Pfister MERL. Outline. Introduction to Multilinear Model Multilinear Face Model Face Transfer. A. I 2. X 1. X 2. X 1 -X 2. X 1. X 2. I 1. B. Y 1. Y 2. Y 1 -Y 2. Y 1. Y 2.

Download Presentation

Face Transfer with Multilinear Models

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Face transfer with multilinear models

Face Transfer with Multilinear Models

Daniel Vlasic & Jovan Popovic

CSAIL MIT

Matthew Brand & Hanspeter Pfister

MERL


Outline

Outline

  • Introduction to Multilinear Model

  • Multilinear Face Model

  • Face Transfer


Face transfer with multilinear models

A

I2

X1

X2

X1-X2

X1

X2

I1

B

Y1

Y2

Y1-Y2

Y1

Y2

X1-Y1

X2-Y2

(X1-Y1) –

(X2-Y2)

X1

X2

1

0

X1

X2

Y1

Y2

=

1

0

x

Y1

Y2

1

-1

X1-Y1

X2-Y2

U(1)

A = (U(2)x(U(1)xB)T)T

1

0

X1

Y1

X1-Y1

AT

1

0

=

x

A =B x1U(1) x2U(2)

X2

Y2

X2-Y2

1

-1

U(2)


Linear model

Linear Model

J2

U(2)

I1

J2

A

I1

U(1)

I2

J1

=

B

I1

J1


Multilinear model

Multilinear Model

  • Generalization of linear model

  • A = B x1U(1)x2U(2)x3U(3)…xnU(n)…xNU(N)

Orthogonal

Transformation

Data Tensor

Core Tensor


How to multiply

How to Multiply?

  • A = B x1 U(1)x2 U(2)x3 U(3)…xn U(n)…

  • B xn U(n) :U(n) * B(n)

I2

Tensor Flattening

X1

X2

I1

B

A =B x1U(1) x2U(2)

Y1

Y2


Tensor flattening

Tensor Flattening


Example

Example

0

0

A(1) =

A=

2

4

1

1

0

2

2

0

2

4

1

-1

2

2

-2

4

1

2

2

0

2

4

0

4

-1

-2

0

0

1

2

1

2

2

4


Face in multilinear model

Face in Multilinear Model

Data Tensor


Mathematically

Mathematically…

?

Data

Tensor

Left Singular of SVD

In data reduction, we use PCA as Y = eTX

  • SVD => A = USVT

    • AAT = USVT(USVT)T = USVT * ((VT)TSUT) = US2UT

  • PCA => Find Cov(A) = (AAT)/(n-1)

    • AAT = eDeT => U = e


  • Svd for multilinear model

    SVD for Multilinear Model

    • To find Un, perform SVD on mode n space of the data tensor, i.e., J(n)

    • This is not optimal, however, and they use ALS, or Alternating Least Square

    • A lot of SIAM papers address this topic, and out of our scope


    Mathematically again

    Mathematically… Again


    Multilinear face model

    Multilinear Face Model

    • Bilinear Model (3-mode)

      • 30K vertices x 10 expression x 15 identities

  • Trilinear Model (4-mode)

    • + 5 visemes

  • Multilinear model of face geometry


    Arbitrary interpolation

    Arbitrary Interpolation

    n

    Synthesized Data, f

    1

    =

    n

    Original Data, M

    m

    Weighting, w

    m rows data

    1

    x

    f= Mx2w(2)


    Interpolation in multilinear model

    Interpolation in Multilinear Model

    F = M x2w(2)

    Multilinear model of face geometry

    f = M x2 w(2) x3 w(3) x4 w(4) …. xN w(N)


    Missing data

    Missing Data

    • So far, we dealt with perfect data set

    • In practice… NOT the case

    • Maximum A Posteriori (MAP) estimation failed

    • Probability Principle Component Analysis (PPCA)


    Short review on ppca

    Short Review on PPCA

    • t = Wx +μ +ε

    • x is N(0, I) , εis isotropic error N(0, σ2I)

    • So t is N(μ, WWT + σ2I)

    • Given t, we want to estimate W, σ

    • Maximize the likelihood function L = p(t) = Πip(ti |x,W)


    Short review on ppca1

    Short Review on PPCA

    • Maximum Likelihood Estimators (M.L.E) tells us that, by taking log-likelihood

    • WML = Uq(Λq – σ2I)1/2R

    • σML = 1/(d-q) Σj = q+1 to dλj

    • Uq is eigen-vector and Λq eigen-value

    • -------------------------------------------------------

    • End of review


    Probabilistic face model

    Probabilistic Face Model

    t = Wx + μ +ε

    Likelihood Function

    p(t |x,W)


    Missing data1

    Missing Data

    Tj = mode-j of J

    =>

    Jj = mode-j of

    Mx2U(2)…xj-1U(j-1) xj+1U(j+1) ..xn U(n)


    Face tracking

    Face Tracking

    • Kanade-Lucas-Tomasi (KLT) algorithm

    • Zd = Z(p – p0) = e

    • Z(sR fi + t - po) = e for vertex i

    • Z(sR Mm,iwm + t – p0) = e


    Comparison

    Comparison


    Result

    Result


  • Login