Modeling the Shape of People from 3D Range Scans

1. Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne Koller, Sebastian Thrun, James Davis

2. The Dataset 70 scans • Cyberware Scans • 4 views, • ~125k polygons • ~65k points each • Problems • Missing surface • Drastic pose changes 48 scans

3. Modeling Human Shape Pose variation Body-shape variation

4. Space of Human Shapes Movie [scape movie]

5. Talk outline • Data processing pipeline • Registration • Recovering an articulated skeleton • Modeling the space of human deformations • Pose deformations • Body shape deformations • Shape completion • Partial view completion • Animating motion capture sequences

6. Talk outline • Data processing pipeline • Registration • Recovering an articulated skeleton • Modeling the space of human deformations • Pose deformations • Body shape deformations • Shape completion • Partial view completion • Animating motion capture sequences

7. Data Processing Pipeline

8. Registration - CC Algorithm [Anguelov et al. 2004] Z X Correlated Correspondence Algorithm • Computes an embedding of mesh Z into mesh X • 1. Defines a discrete Markov Net Mover correspondence variables of mesh Z • Markov Net potentials enforce • Minimal surface deformation • Similar local surface appearance • Preservation of geodesic distance • 2. Embedding of Z into X is computed by performing Loopy Belief Propagation on M Input: Pair of scans Output: Correspondences X Z Related work: [Huttenlocher et al 00][Coughlan 02]

9. Results: Human poses dataset Cyberware scans Model Registrations • 4 markers were used on each scan to avoid the need for multiple initializations of Loopy-BP

10. Recovering articulation Input: models, correspondences Output: rigid parts, skeleton

11. Recovering articulation [Anguelov et al’04] 4 3 2 1 • Stages of the process • Register meshes using Correlated Correspondences algorithm • Initialize • break template surface into N arbitrary components • Cluster surface into rigid parts • Estimate joints Related work: [Cheung et al’03]

12. a1 aN Part labels … Transformations Model xN T x1 Points … Transformed Model y1 yN … b1 bK Point corrs Instance z1 zK Points … Probabilistic Generative Model

13. Contiguity Prior • Parts are preferably contiguous regions • Adjacent points on the surface should have similar labels • Enforce this with a Markov network: a1 a2 a3

14. Clustering algorithm • Objective • Algorithm • Given transformations , perform min-cut* inference to get • Given labels , solve for rigid transformations Data Likelihood Contiguity prior * [Greig et al. 89], [Kolmogorov & Zabih 02]

15. Results: Puppet articulation

16. Results: Arm articulation

17. Results: 50 scans of a human Tree-shaped skeleton found Rigid parts found

18. Talk outline • Data processing pipeline • Registration • Recovering an articulated skeleton • Modeling the space of human deformations • Pose deformations • Body shape deformations • Shape completion • Partial view completion • Animating motion capture sequences

19. Modeling Pose Deformation input Joint angles Deformations output Regression function

20. Modeling Pose Deformation Predict independently for each triangle Reconstruct complete shape template Related work: [Allen et al ‘03][Sumner+Popovic 2004]

21. Rigid articulated deformation Pose deformation Given estimates of R, Q, synthesizing the shape is straightforward : Representation of pose deformation

22. Learning pose deformation • For each polygon, predict entries of from rotations of nearest 2 joints (represented as twists ). • Linear regression parameters : • Obtaining values of in the first place:

23. Twists and exponential maps Twist From twist to rotation matrix Joint angles

24. Pose deformation space

25. Learning body-shape deformation • Include also change in shape due to different people: • Do PCA over body-shape matrices : • Getting estimates of :

26. PCA over body shape

27. Combining pose and body shape spaces

28. Talk outline • Data processing pipeline • Registration • Recovering an articulated skeleton • Modeling the space of human deformations • Pose deformations • Body shape deformations • Shape completion • Partial view completion • Animating motion capture sequences

29. Shape completion • Find surface Y from our space which matches a set of markers Z • Y[Z] : completed mesh • deforms out of space spanned by , R to match Z • Y’[Z]: predicted mesh • constrained to be in space spanned by , R • Target optimized by iteratively solving for , R orY while holding the others fixed

30. Partial view completion • Process: • Add a few markers (~6-8) • Run CC algorithm to get > 100 markers • Optimize to find Y[Z]

31. Shape completion from motion capture data

32. Conclusions • Presented a data-driven method of modeling human deformations induced by • Pose • Body shape • Extending the model • Nonlinear prediction of pose deformation (ongoing) • Shape complete original scans using current model • Acquire and learn from the entire pose-bodyshape matrix • Prior on likely joint angles, e.g. [Popovic + et al ’04] • Enforce temporal consistency in tracking applications • Extending the possible applications • Markerless motion capture (shape completion in shape-from-silhouette data) • Modeling other beasts