vision based control of 3d facial animation n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Vision-based Control of 3D Facial Animation PowerPoint Presentation
Download Presentation
Vision-based Control of 3D Facial Animation

Loading in 2 Seconds...

play fullscreen
1 / 24

Vision-based Control of 3D Facial Animation - PowerPoint PPT Presentation


  • 354 Views
  • Uploaded on

Vision-based Control of 3D Facial Animation. Jin-xiang Chai Jing Xiao Jessica Hodgins Carnegie Mellon University. Our Goal. Interactive avatar control Designing a rich set of realistic facial actions for a virtual character Providing intuitive and interactive control over these actions.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Vision-based Control of 3D Facial Animation' - Ava


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
vision based control of 3d facial animation

Vision-based Control of 3D Facial Animation

Jin-xiang Chai

Jing Xiao

Jessica Hodgins

Carnegie Mellon University

our goal
Our Goal

Interactive avatar control

  • Designing a rich set of realistic facial actions for a virtual character
  • Providing intuitive and interactive control over these actions
control interface vs quality

+Inexpensive

+ Non-intrusive

-Noisy

- Low resolution

- Expensive

- Intrusive

+ High quality

Control Interface vs. Quality

Control Interface

Quality

Vision-based animation

Online motion capture

our idea
Our Idea

+

Vision-based interface

Motion capture database

Interactive avatar control

related work
Related Work

Motion capture

  • Making faces [Guenter et al. 98]
  • Expression Cloning [Noh and Neumann 01]

Vision-based tracking for direct animation

  • Physical markers [Williams 90]
  • Edges [Terzopoulos and Waters 93, Lanitis et al. 97]
  • Dense optical flow with 3D models [Essa et al. 96, Pighin et al. 99, DeCarlo et al. 00]
  • Data-driven feature tracking [Gokturk et al. 01]

Vision-based animation with blendshape

  • Hand-drawn expression [Buck et al. 00]
  • 3D model avatar model [FaceStation]
system overview

Video Analysis

System Overview

Expression

Video

Preprocessed

Control and

Analysis

Motion Capture

Animation

Data

Act out expressions

Expression

Retargeting

Avatar animation

video analysis
Video Analysis
  • Vision-based tracking
    • 3D Head Poses [Xiao et al. 2002]
    • 2D facial features

Video

Analysis

expression control parameters
Expression Control Parameters

Extracting 15 expression control parameters from 2D tracking points

Distance between two feature points

Distance between a point and a line

Orientation and center of the mouth

t

Expression control signal

system overview1
System Overview

Expression

Video

Control and

Preprocessed

Analysis

Animation

Motion Capture

Data

Act out expressions

Expression

Retargeting

Avatar animation

motion capture data preprocessing
Motion Capture Data Preprocessing

Expression separation

3D Poses

  • 70000 frames (10 minutes) including:
  • 6 basic facial expressions
  • typical everyday facial expressions
  • speech data

Expression control parameter extraction

system overview2
System Overview

Expression

Video

Control and

Preprocessed

Analysis

Animation

MotionCapture

Data

Act out expression

Expression

Retargeting

Avatar animation

expression control

Expression control parameters

Expression control parameters

15 dofs

15 dofs

Expression Control

19*2 dofs

3D motion data

2D tracking data

76*3 dofs

Vision-based interface

Motion capture database

challenges
Challenges
  • Visual expression control signals are very noisy
  • One to many mapping from expression control parameter space to 3D motion space

15 dofs

76*3 dofs

Temporal coherence

Control parameter space

3D motion space

data driven dynamic filtering
Data-driven Dynamic Filtering

Noisy control signal

K=120 closest examples

Nearest Neighbor Search

W = 0.33s

Online PCA

Filter by eigen-curves

7 largest Eigen-curves (99.5 % energy)

Filtered control signal

expression mapping
Expression Mapping

From expression control parameter space to 3D motion data space

d1

w(d1)

Synthesized motion

Filtered control signal

d2



w(d2)

Nearest Neighbor Search

...

...

dK

w(dK)

system overview3

Expression

Retargeting

System Overview

Expression

Video

Control and

Preprocessed

Analysis

Animation

Motion Capture

Data

Act out expression

Avatar animation

expression retarget
Expression Retarget

Synthesized expression

Avatar expression

expression retarget1

xs

xt

?

xs

xt

Expression Retarget
  • Learn the surface mapping function using Radial Basis Functions such that xt=f(xs)
  • Transfer the motion vector by local Jacobian matrix Jf(xs) by xt=Jf(xs) xs

Run time computational cost depends on the number of vertices

precompute deformation basis
Precompute Deformation Basis

PCA

25 source motion bases –99.5% energy

S0

S1

S2

S3

S4

S5

Precompute deformation basis

25 precomputed avatar motion bases

T0

T2

T3

T4

T5

T1

target motion synthesis
Target Motion Synthesis

Synthesized expression

iSi

S0

S1

S2

S3

SN

0,….N

iTi

T2

T0

T1

T3

TN

Avatar expression

Run time computational cost is O(N)

N is the number of bases

system overview4

Expression Retargeting

System Overview

Expression

Video

Control and

Preprocessed

Analysis

Animation

Motion Capture

Data

Act out expression

Avatar animation

conclusions
Conclusions
  • Developed a performance-based facial animation system for interactive expression control
    • Tracking real-time facial movements in video
    • Preprocessing the motion capture database
    • Transforming low-quality 2D visual control signal to high quality 3D facial expression
    • An efficient online expression retarget
future work
Future Work
  • Formal user study on the quality of the synthesized motion
  • Controlling and animating 3D photorealistic facial expression
  • Size of database