1 / 24

Vision-based Control of 3D Facial Animation

Vision-based Control of 3D Facial Animation. Jin-xiang Chai Jing Xiao Jessica Hodgins Carnegie Mellon University. Our Goal. Interactive avatar control Designing a rich set of realistic facial actions for a virtual character Providing intuitive and interactive control over these actions.

ion
Download Presentation

Vision-based Control of 3D Facial Animation

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Vision-based Control of 3D Facial Animation Jin-xiang Chai Jing Xiao Jessica Hodgins Carnegie Mellon University

  2. Our Goal Interactive avatar control • Designing a rich set of realistic facial actions for a virtual character • Providing intuitive and interactive control over these actions

  3. +Inexpensive + Non-intrusive -Noisy - Low resolution - Expensive - Intrusive + High quality Control Interface vs. Quality Control Interface Quality Vision-based animation Online motion capture

  4. Our Idea + Vision-based interface Motion capture database Interactive avatar control

  5. Related Work Motion capture • Making faces [Guenter et al. 98] • Expression Cloning [Noh and Neumann 01] Vision-based tracking for direct animation • Physical markers [Williams 90] • Edges [Terzopoulos and Waters 93, Lanitis et al. 97] • Dense optical flow with 3D models [Essa et al. 96, Pighin et al. 99, DeCarlo et al. 00] • Data-driven feature tracking [Gokturk et al. 01] Vision-based animation with blendshape • Hand-drawn expression [Buck et al. 00] • 3D model avatar model [FaceStation]

  6. Video Analysis System Overview Expression Video Preprocessed Control and Analysis Motion Capture Animation Data Act out expressions Expression Retargeting Avatar animation

  7. Video Analysis • Vision-based tracking • 3D Head Poses [Xiao et al. 2002] • 2D facial features Video Analysis

  8. Expression Control Parameters Extracting 15 expression control parameters from 2D tracking points Distance between two feature points Distance between a point and a line Orientation and center of the mouth t Expression control signal

  9. System Overview Expression Video Control and Preprocessed Analysis Animation Motion Capture Data Act out expressions Expression Retargeting Avatar animation

  10. Motion Capture Data Preprocessing Expression separation 3D Poses • 70000 frames (10 minutes) including: • 6 basic facial expressions • typical everyday facial expressions • speech data Expression control parameter extraction

  11. System Overview Expression Video Control and Preprocessed Analysis Animation MotionCapture Data Act out expression Expression Retargeting Avatar animation

  12. Expression control parameters Expression control parameters 15 dofs 15 dofs Expression Control 19*2 dofs 3D motion data 2D tracking data 76*3 dofs Vision-based interface Motion capture database

  13. Challenges • Visual expression control signals are very noisy • One to many mapping from expression control parameter space to 3D motion space 15 dofs 76*3 dofs Temporal coherence Control parameter space 3D motion space

  14. Data-driven Dynamic Filtering Noisy control signal K=120 closest examples Nearest Neighbor Search W = 0.33s Online PCA Filter by eigen-curves 7 largest Eigen-curves (99.5 % energy) Filtered control signal

  15. Expression Mapping From expression control parameter space to 3D motion data space d1 w(d1) Synthesized motion Filtered control signal d2  w(d2) Nearest Neighbor Search ... ... dK w(dK)

  16. Expression Retargeting System Overview Expression Video Control and Preprocessed Analysis Animation Motion Capture Data Act out expression Avatar animation

  17. Expression Retarget Synthesized expression Avatar expression

  18. xs xt ? xs xt Expression Retarget • Learn the surface mapping function using Radial Basis Functions such that xt=f(xs) • Transfer the motion vector by local Jacobian matrix Jf(xs) by xt=Jf(xs) xs Run time computational cost depends on the number of vertices

  19. Precompute Deformation Basis PCA 25 source motion bases –99.5% energy … S0 S1 S2 S3 S4 S5 Precompute deformation basis … 25 precomputed avatar motion bases T0 T2 T3 T4 T5 T1

  20. Target Motion Synthesis Synthesized expression iSi … S0 S1 S2 S3 SN 0,….N … iTi T2 T0 T1 T3 TN Avatar expression Run time computational cost is O(N) N is the number of bases

  21. Expression Retargeting System Overview Expression Video Control and Preprocessed Analysis Animation Motion Capture Data Act out expression Avatar animation

  22. Results

  23. Conclusions • Developed a performance-based facial animation system for interactive expression control • Tracking real-time facial movements in video • Preprocessing the motion capture database • Transforming low-quality 2D visual control signal to high quality 3D facial expression • An efficient online expression retarget

  24. Future Work • Formal user study on the quality of the synthesized motion • Controlling and animating 3D photorealistic facial expression • Size of database

More Related