Modeling the Shape of People from 3D Range Scans

1. Modeling the Shape of People from 3D Range Scans Dragomir Anguelov AI Lab Stanford University Joint work with Praveen Srinivasan, Hoi-Cheung Pang, Daphne Koller, Sebastian Thrun

2. The Dataset • Scans • 4 views, • ~125k polygons • ~65k points each • Problems • Missing surface • Drastic pose changes

3. Space of Human Shapes movie [scape movie]

4. The Modeling Pipeline

5. Talk outline • Scan registration • Rigid • Nonrigid • Correlated correspondence • Decomposition into approximately rigid parts • Learning a deformable human model

6. Talk outline • 3D Scan registration • Rigid • Nonrigid • Correlated correspondence • Decomposition into approximately rigid parts • Learning a deformable human model

7. Rigid Scan Registration • Aligning point clouds X and Z • Match each point in Z to its closest point in X • Compute the best rigid transformation T which aligns the matching pairs • Transform the points of cloud X by T • Repeat • Iterative Closest Point (ICP) algorithm [Besl et al ’92]

8. ICP matching step • Objective • Setting obtains • Substituting the above for s yields a system linear in R • Choose a parameterization of R (quaternions q) and solve • Exact solution • Overall, ICP converges to a local minimum of the energy (using )

9. Deformable templatemodel X Set of deformable springs Nonrigid registration Deforms X to match a scan Z Minimizes the deformation of X Nonrigid Registration Example

10. Data Generation / Correspondences Deformation / Transformation C Q Model mesh X Transformed mesh X’ Data mesh Z Generative Model Goal: Given meshes X and Z, recover transformation Q and correspondences C

11. Data mesh generation Transformed Model Data Correspondence ck specifies which point in X’ generates point zk: Z X’

12. X Z Nonrigid ICP failure example • Nonrigid ICP attempts to maximize • Local Minima • Poor transformation and correspondence estimates reinforce each other

13. Nonrigid-ICP decorrelates the correspondences Correspondences are conditionally independent given Q If Q is unknown, correspondences are correlated E.g. nearby points in Z should be mapped to nearby points in X The search space is exponential for the general registration problem c1 c2 Correlated correspondences

14. Correlated Correspondence • Compute an embedding of mesh Z into model X • The domain of ciis the set of possible (discrete) matches for zion the surface of X • The solution is a consistent assignment to all correspondence variables C Z X

15. (c1) (c2) (c1, c2) c1 c2 (c1,c3) (c2,c3) c3 (c3) Correlated Correspondence Z • Edge is generated from some edge in X • Prefer values of c1, c2 according to the deformation of X they induce • Doing this for all edges in Z results in a Markov network over C X

16. {1, …, K} Pairwise Markov Networks ij • Inference is Markov Nets is generally intractable • Good approximate algorithms exist (belief propagation) i yi yj

17. (c1) (c2) (c1, c2) c1 c2 (c1,c3) (c2,c3) c3 (c3) CC Potentials • Deformation potentials • Spin image* potentials • Quantify the similarity of the local surface around two matching points • Geodesic Potentials *[Johnson+Hebert ’97]

18. Nearby points in Z must be nearby in X Constraint between each pair of adjacent points zk, zl Distant points in Z must be distant in X Constraint between each pair of distant points zk, zl (farther than 5r) Geodesic potentials Z X X Z r resolution of mesh X

19. Local surface signatures • Use spin-images[Johnson ’97] • 2D Histogram of distances from an oriented reference point • Rotationally-invariant / Robust under clutter and occlusion / Compressible (PCA) • Potentialf(ck = i) encodes how well the signature of point zk matches the signature of point xi in the model:

20. Results: Human poses dataset Cyberware scans Model Registrations • 4 markers were used on each scan to avoid the need for multiple initializations of Loopy-BP • ArtModel I found using 20 registered scans • ArtModel II found using 44 registered scans ArtModel II ArtModel I

21. CC movie [movie]

22. Talk Outline • Scan registration • Rigid • Nonrigid • Correlated correspondence • Recovering the articulated skeleton • Learning a deformable human model

23. Recovering the skeleton Input: models, correspondences Output: rigid parts, skeleton

24. … Part labels a1 aN Transformations Model … xN T Points x1 Transformed Model … y1 yN b1 bK Point labels Instance … z1 zK Points bk = ModelPoint (zk) Probabilistic Generative Model aj = Part (xj)

25. a1 a3 If i, j are connected in model mesh a2 1 >  > 0.5 Part Properties • Parts are preferably contiguous regions • Adjacent points on the surface should have similar labels • Enforce this with a Markov network:

26. Iterative Optimization • Hard EM • Given transformations , perform min-cut inference to get • Given labels , solve for rigid transformations

27. Results: Puppet articulation

28. Results: Arm articulation

29. Parts movie [parts.gif]

30. Talk Outline • Scan registration • Rigid • Nonrigid • Correlated correspondence • Recovering the articulated skeleton • Learning a deformable human model

31. Modeling Human Shape Pose variation Body-shape variation

32. Representation of pose deformation Rigid articulated deformation Pose deformation Given estimates of R, Q, synthesizing the shape is straightforward :

33. Learning non-rigid pose deformation • For each polygon, predict entries of from rotations of nearest 2 joints (represented as twists ). • Linear regression parameters :

34. Learning body-shape deformation • Include also change in shape due to different people: • Do PCA over body-shape matrices :

35. Shape completion • Process: • Add a few markers (~6-8) • Run CC algorithm to get >100 markers • Optimize for pose/body-shape parameters and non-rigid surface

36. Mocap shape completion

37. Recap • Algorithms for registration between two surfaces • ICP algorithm iterates between finding corresponding points between two scans and computing the transformation between them • Rigid variant (6 unknown DOF) • Nonrigid variant (6  Number of polygons DOF) • CC explicitly encodes correlations between correspondence variables • Essential when we have poor initial transformation estimate • Exponential complexity, approximate algorithm • Modeling humans • Decouple effects of pose and different physique on shape deformation

38. The end

39. Geodesic Potentials: closeclose • Nearby points in Zmust be nearby in X • Constraint between each pair of adjacent points zk, zl Z X

40. X Z Geodesic Potentials: farfar • Distant points in Zmust be distant in X • Constraint between each pair of distant points zk, zl