face poser interactive modeling of 3d facial expressions using model priors
Download
Skip this Video
Download Presentation
Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors

Loading in 2 Seconds...

play fullscreen
1 / 49

Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors - PowerPoint PPT Presentation


  • 128 Views
  • Uploaded on

Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors. Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3. 1 Carnegie Mellon University 2 Texas A&M University 3 Microsoft Research Asia. Face Poser.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors' - octavia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
face poser interactive modeling of 3d facial expressions using model priors

Face Poser:Interactive Modeling of 3D Facial Expressions Using Model Priors

Manfred Lau1,3 Jinxiang Chai2 Ying-Qing Xu3 Heung-Yeung Shum3

1Carnegie Mellon University 2Texas A&M University 3Microsoft Research Asia

face poser
Face Poser

Generate new facial expressions with a simple and intuitive interface

Inputs

face poser3
Face Poser

Generate new facial expressions with a simple and intuitive interface

Inputs

Output

why face poser
Why Face Poser?

Complex facial expressions

Pre-defined controls

Difficult to build and use

applications
Applications

Films, Games

Virtual Reality

Educational

related work
Related Work

Sketched-based interfaces

Zeleznik et al. 96

Igarashi et al. 99

Nealen et al. 05

Kho and Garland 05

Chang and Jenkins 06

Igarashi et al. 99

Nealen et al. 05

related work7
Related Work

Example-based modeling

Blanz and Vetter 99

Pighin et al. 99

Chai et al. 03

Zhang et al. 04

Grochow et al. 04

Sumner et al. 05

Grochow et al. 04

Sumner et al. 05

overview
Overview

Database

Preprocessing

Model Prior

overview9
Overview

Database

Preprocessing

Model Prior

User Constraints

Neutral Pose

Interface

overview10
Overview

Database

Preprocessing

Model Prior

RuntimeOptimization

User Constraints

Neutral Pose

New Pose

Textured Pose

Interface

motion capture data
Motion capture data

Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing

All meshes translated and rotated to a standard view:

data pca representation
Data: PCA representation

v1x

v1y

v1z

v2x

.

.

.

x =

p is low-dimensional representation of x

problem statement
Problem statement

Find best p satisfying user-constraints c:

Best p is:

Given a face model, how well does it match user-constraints

Likelihood of face model using knowledge of data

point constraints
Point Constraints

More detailed control

User inputs:

blue – 3D source vertex

green – 2D target pixel

Can select in any camera view

point constraints15
Point Constraints

We optimize for best p

For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel

point constraints16
Point Constraints

Optimization term:

Jacobian term:

point constraints17
Point Constraints

Inputs

Solution

stroke constraints
Stroke Constraints

Large-scale changes with minimal input

User inputs:

blue – 2D source stroke (selects 3D points on mesh)

green – 2D target stroke

Any curve, line, or freeform region

stroke constraints21
Stroke Constraints

2D source stroke  raytrace each pixel to mesh surface to get dark gray points

These can be 3D points on mesh surface (not just original mesh vertices)

stroke constraints22
Stroke Constraints

We optimize for best p

For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke

stroke constraints23
Stroke Constraints

Optimization term:

Jacobian term:

stroke constraints24
Stroke Constraints

Inputs

Solution

stroke constraints additional term
Stroke Constraints – Additional term

If strokes are far away from each other, energy term will reach local minimum

Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke

Without additional term

stroke constraints additional term28
Stroke Constraints – Additional term

Optimization term:

Jacobian term:

Without additional term

stroke constraints additional term29
Stroke Constraints – Additional term

Without additional term

With additional term

problem statement31
Problem statement

Find best p satisfying user-constraints c:

Best p is:

Given a face model, how well does it match user-constraints

Likelihood of face model using knowledge of data

model priors
Model Priors

There can be many solutions satisfying user constraints. Some of them are not realistic.

We add another optimization term to constrain the solution to the space defined by the motion capture data.

Without model priors term

model priors33
Model Priors

Learn a Mixtures of Factor Analyzers (MFA) model

Probability density function to measure naturalness of facial expression

MFA has been applied to high-dimensional nonlinear data

Without model priors term

model priors34
Model Priors

Optimization term:

Jacobian term:

Without model priors term

model priors result36
Model Priors – Result

increasing weight on Model Prior term

computation time
Computation time

Standard PC hardware (Pentium IV 2 GHz)

Point constraintstakes 0.18 seconds for 10 points

time increases linearly with number of points

Stroke constraintstakes 0.4 seconds for source stroke of ~900 pixels (about size of eyebrow)

time increases linearly with number of pixels

faster if using intermediate spline representation

cross validation
Cross validation

New face expression samples for testing

Use new samples to get target constraints

Generate solution and compare with test sample

cross validation40
Cross validation

Ground truth Interpolation Optimization

comparison with other techniques
Comparison with other techniques

Opt-blend: FaceIK [Zhang et al. 04]

PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03]

LWR: Locally weighted regression

3D errors

comparison with other techniques42
Comparison with other techniques

Ground truth, Optimization with PCA, Optimization with MFA

application trajectory keyframing
Application: Trajectory Keyframing

Green points – given 2D target pixels

Blue points and mesh – solution

summary face poser
Summary: Face Poser

Users can learn to use our system within minutes and can create new facial expressions within seconds

Inputs

Output

limitation
Limitation

Global control

  • changing mouth also changes eyes
  • this is natural, but difficult to control sometimes

Local control

  • changing mouth without changing eyes
  • but this might lead to “fake smiles”
extensions future work
Extensions / Future work

We have added different types of constraints within the same optimization framework

More general: model face as separate regions, generate each region separately, and blend them back together

ad