Dynamic Mesh Compression with Skinning Approach: State-of-the-Art and Innovations

1. CASA 2006 A Skinning Approach for Dynamic Mesh Compression Khaled Mamou Titus Zaharia Françoise Prêteux

2. 3D animation industry Context & Objectives • Applications Virtual and augmented reality Cartoons Video games and CGI films

3. Dynamic 3D content Context & Objectives • Content creation Motion capture Skinning models Physical-based simulation … How to exchange, transmit and visualize such 3D content in a platform-independent manner ?!

4. Dynamic 3D content Interpolation Interpolation Context & Objectives 3D animation industry: key-frame representations • Principle Represent the animation sequence as a set of key-meshes Key-frames Apply interpolation procedures to generate the in-between frames at the desired framerate Animation

5. Dynamic 3D content Time-varying geometry Constant topology Context & Objectives Key-frame representations: dynamic 3D meshes • Sequence of meshes with: Constant topology Time-varying geometry • Advantages Generality Interoperability Content protection • Drawbacks Huge amount of data • Need of compact representations

6. Objectives • Compression efficiency Compactness of the coded representation • Progressive transmission Bitstream adaptation to different, fixed or mobile communication networks and terminal devices • Scalable rendering Bitstream adaptation for real-time rendering Context & Objectives

7. Previous work Skinning-based compression Experimental evaluation Conclusion et perspectives Outline

8. Previous work Skinning-based compression Experimental evaluation Conclusion et perspectives Outline

9. State of the art Vertex prediction Wavelets Dynamic 3D mesh compression PCA-based Clustering AWC • Emerging field of research GV MPEG-4/AFX-IC Dynapack • Four families of approaches PCA RT LPCA D3DMC CPCA

10. State of the art Vertex prediction Wavelets Dynamic 3D mesh compression PCA-based Clustering Skinning-based compression AWC • Principle: extension of the RT technique GV MPEG-4/AFX-IC Dynapack A more elaborated motion model: skinning New motion-based segmentation procedure PCA Temporal DCT-based compression of the residual errors RT LPCA D3DMC CPCA

11. Previous work Skinning-based compression Experimental evaluation Conclusion et perspectives Outline

12. General view Skinning-based compression Static encoder Compressed M0 Prediction residuals Compressed DCT coefficients M0 Temporal DCT Affine motion and weights estimation Affine transforms Animation weights Quantization and arithmetic encoding (Mi) Motion-based segmentation Partition

13. Motion-based segmentation Skinning-based compression • Objective Partition the mesh vertices into clusterswhose motion can be accurately described by a single affine motion

14. Motion-based segmentation Skinning-based compression • Principle For each vertex v, select a neighborhood v*

15. Motion-based segmentation Vector of homogeneous coordinates of vertex pat frame i Skinning-based compression • Principle For each vertex v, select a neighborhood v* … For each frame i, compute an affine transform Aiv Store the (Aiv)i of each vertex as a single vector αv Frame 0 Frame 1 Frame (F-1)

16. Motion-based segmentation Skinning-based compression • Principle For each vertex v, select a neighborhood v* For each frame i, compute an affine transform Aiv Store the (Aiv)i of each vertex as a single vector αv Determine the partition π= (πk)k by applying the k-means clustering algorithm to the set (αv)v Cow

17. Motion-based segmentation Skinning-based compression • Principle For each vertex v, select a neighborhood v* For each frame i, compute an affine transform Aiv Store the (Aiv)i of each vertex as a single vector αv Determine the partition π= (πk)k by applying the k-means clustering algorithm to the set (αv)v Dancer

18. General view Skinning-based compression Static encoder Compressed M0 Prediction residuals Compressed DCT coefficients M0 Temporal DCT Affine motion and weights estimation Affine transforms Animation weights Quantization and arithmetic encoding (Mi) Motion-based segmentation Partition

19. Affine motion estimation Skinning-based compression • Principle Model the motion of each cluster k at each frame i by an affine transform Hik Predict the geometry of frame i from frame 0 by using the affine transforms(Hik)k

22. Affine motion estimation 4% 0% Frame 0 Frame 36 Predicted frame 36 Error distribution Skinning-based compression • Performances Captures well the object motion Induces discontinuities at the level of clusters boundaries We need a more elaborated motion model

23. Skinning model Skinning-based compression • Objective Derive a continuous motion field • Principle Linearly combine the affine motion of adjacent clusters with appropriate weighting coefficients Compute the animation weights by solving a least squares minimization problem

26. General view Skinning-based compression Static encoder Compressed M0 Prediction residuals Compressed DCT coefficients M0 Temporal DCT Affine motion and weights estimation Affine transforms Animation weights Quantization and arithmetic encoding (Mi) Motion-based segmentation Partition

27. DCT-based compression of the residual errors Skinning-based compression • Objective Prediction error at frameiand vertex v Compress the residual errors by exploiting the temporal correlations • Principle For each vertex v, compute the spectra of its x, y and z errors Concatenate the spectral coefficients of all vertices into a single vector S Quantize and arithmetically encode S Well-adapted to progressive transmission

28. Previous work Skinning-based compression Experimental evaluation Conclusion et perspectives Outline

29. Experimental results Evaluation corpus: Snake 9179 vertices 134 frames

30. Experimental results Evaluation corpus: Dancer 7061 vertices 201 frames

31. Experimental results Evaluation corpus: Humanoid 7646 vertices 154 frames

32. Experimental results Evaluation corpus: Chicken 3030 vertices 400 frames

33. Experimental results Objective evaluation: criteria • Compression rates: bits per frame per vertex (bpfv) • Distortion measures: RMSE [MESH tool, Aspert et al, 2002] D: length of the diagonal of the object’s bounding box

34. Experimental results Compression results: Chicken RMSE • Performances D3DMC & skinning: best performances Skinning: up to 47% gain over D3DMC in term of bitrates bpfv

35. Experimental results Compression results: Snake RMSE • Performances PCA: worst performances (F>>V not verified) Skinning: up to 45% gain over RT in term of bitrates bpfv

36. Experimental results Compression results: Humanoid RMSE • Performances AFX-IC: poor performances: elementary predictor Skinning: up to 67% gain over D3DMC in term of bitrates bpfv

37. Experimental results Compression results: Dancer RMSE • Performances GV: re-meshing related problems Skinning: up to 65% gain over GV in term of bitrates bpfv

38. Previous work Skinning-based compression Experimental evaluation Conclusion et perspectives Outline

39. Conclusion & perspectives Summary • A new skinning-based compression techniques for dynamic meshes • Specifically efficient for articulated dynamic meshes • Gains range from 47% to 67% in terms of bitrates over state-of-the-art encoders

40. Conclusion & perspectives Future work • Optimize the motion-based segmentation stage: How to determine automatically the number of clusters? • Multiple and dynamic skinning models: Temporal segmentation of the sequence • Compression of other attributes: normals, texture coordinates…

41. Thank you