Chap 6 Motion Capture (Mocap)

1. Chap 6Motion Capture (Mocap)

2. What is motion capture? • Key-frames, forward kinematics, inverse kinematics • A daunting task for creating physically realistic motion • Requires a large amount of animation talent • Record the motion and map it into a synthetic object • Much easier • Realistic motion CS, NCTU, J. H. Chuang

3. What is motion capture? • track motion of reference points • convert to joint angles to drive • an articulated 3D model • a deformable surface CS, NCTU, J. H. Chuang

4. What is motion capture? • Objective • Reconstruct the 3D motion of a physical object and apply it to a synthetic object • Recording 3D live action, including • whole body • face • hands • animals CS, NCTU, J. H. Chuang

5. Applications • Animation • Special effects • Robot control • Interactive characters • Games CS, NCTU, J. H. Chuang

6. Computer Puppetry Shin et al., “Computer puppetry” CS, NCTU, J. H. Chuang

7. VideoOptical Motion Capture in Games CS, NCTU, J. H. Chuang

8. Video: Lord of the rings: Gollum CS, NCTU, J. H. Chuang

9. Video • Facial Mocap in King Kong CS, NCTU, J. H. Chuang

10. Video: Irobot CS, NCTU, J. H. Chuang

11. Pros and Cons of Motion Capture • Pros • All fine details of human motion will be recorded-- if they can be captured • Cons • Not so easy to • Edit • Generalize • Control • Not cheap CS, NCTU, J. H. Chuang

12. What is captured? • What do we need to know? • X,Y,Z • roll, pitch, yaw • Errors cause • Joints to come apart • Links to grow/shrink • Bad contact points CS, NCTU, J. H. Chuang

13. What is captured? • Large and small scale CS, NCTU, J. H. Chuang

14. How to use the data? • Off-line • Processed by filtering, inverse kinemtics • Produce libraries of motion trajectories • Choose among them • Blend between them • Modify on the fly • On-line (performance animation) • Driving character directly based on what actor does in real time CS, NCTU, J. H. Chuang

15. History of the technology • Recording motion for biomechanics • High accuracy • Fewer recorded pints • Hand digitizing film • Supplement with force plate, muscle activity • Computer animation • Rotoscoping • VR tracking technology • Less accuracy required • Fewer sensors CS, NCTU, J. H. Chuang

16. Optical Motion Capture (Passive) • Passive reflection • Camera • Infrared, visible, or near infrared strobes • High resolution (1 to 4 million pixels) • 120-240 frames/sec (max 2000 frames/sec) • Not outdoors • No glossy or reflective materials • Tight clothing • Occlusion of markers by limbs or props CS, NCTU, J. H. Chuang

17. Optical Motion Capture (Active) • Active output of the LED • No marker confusion problem • Outdoor capture • 120 frames/sec (128 markers or four persons) • 480 frames/sec (32 markers or single person) • 1/3 the cost of passive systems CS, NCTU, J. H. Chuang

18. Magnetic Motion Capture • Electromechanical transducer • Heavier sensors • Wires on body (wireless back to base station) • Limited accuracy (~10x less accuracy than optical) CS, NCTU, J. H. Chuang

19. Magnetic Motion Capture (cont.) • Smaller workspace • Sensors are the cost • Sensitive to EMI/metal • Relatively cheaper than optical device Ascension MotionStar Wireless CS, NCTU, J. H. Chuang

20. Mechanical Motion Capture • Subject wears an exoskeleton • No interference from light or magnetic field • No marker confusions • No range limit • Some restriction of movement • Absolute position is unknown CS, NCTU, J. H. Chuang

21. Mechanical Motion Capture • Data glove • Bend sensor + optical tracking • 6 DOF • video http://www.vrealities.com/glove.html CS, NCTU, J. H. Chuang

22. Technology Issues • Resolution/range of motion • Calibration • Accuracy • Marker movement • Sensor noise • Restrictions on the environment • Capture rate • Occlusion/correspondence CS, NCTU, J. H. Chuang

23. Markers - Examples CS, NCTU, J. H. Chuang

24. Skin Motion CapturePark & Hodgins, SIGGRAPH’06 • Uses a conventional optical motion capture system 40-60 markers CS, NCTU, J. H. Chuang

25. A dense set of markers Skin Motion Capture • Uses a conventional optical motion capture system CS, NCTU, J. H. Chuang

26. A dense set of markers Skin Motion Capture • Uses a conventional optical motion capture system CS, NCTU, J. H. Chuang

27. A dense set of markers Detailed surface model Skin Motion Capture • Uses a conventional optical motion capture system CS, NCTU, J. H. Chuang

28. A dense set of markers Detailed surface model Skin Motion Capture • Uses a conventional optical motion capture system Data collection and cleaning CS, NCTU, J. H. Chuang

29. A dense set of markers Detailed surface model Skin Motion Capture • Uses a conventional optical motion capture system Data collection and cleaning Skin Animation CS, NCTU, J. H. Chuang

30. What is motion capture?Steps after motion is captured • Process 2D images to locate, identify, and correlated the markers in multiple video streams • Requires image processing techniques • Reconstruct the 3D locations of the markers • Requires camera calibration • Overcome numerical inaccuracies • Constrain the 3D marker locations to a model of physical system whose motion is being captured • Require satisfying constraints between relative marker positions CS, NCTU, J. H. Chuang

31. Camera calibration • Find the locations and orientations of cameras in world space and the intrinsic properties of the camera such as focal length, image center, and aspect ratio • Use a simple pinhole camera model • An idealized model • Usually sufficient for graphics and animation CS, NCTU, J. H. Chuang

32. Camera calibration Pinhole camera model CS, NCTU, J. H. Chuang

33. 3D position reconstruction • Locate a marker’s position in at least two views relative to known camera positions CS, NCTU, J. H. Chuang

34. 3D position reconstruction • Locate a marker’s position in at least two views relative to known camera positions CS, NCTU, J. H. Chuang

35. Fitting to the skeleton • After the motion of the individual markers looks smooth and reasonable, • Attach markers to the underlying skeletal structure that is to be controlled by the digitized motion • The position of each marker is used to absolutely position a specific joint • Is not straight due to noise, smoothing, and inaccuracy • Change in bone length can be significant • Markers are located outside the joints at the surface CS, NCTU, J. H. Chuang

36. Fitting to the skeleton • Given marker location and the relative distance from marker to the joint is not sufficient since the direction of displacement is not known • Put markers on both side of the joint • Work fine for simple joints, but not the complex joints (such as shoulder and spine) • Use normal of a plane formed by 3 markers • Wrist-elbow-shoulder markers plane normal • Offset the elbow marker in the direction of plane normal by the amount measured from the performer CS, NCTU, J. H. Chuang

37. Manipulating motion capture data • Processing the signal • Motion signal processing and Motion warping • Considers how frequency components capture various qualities of the motion. And warps the signal in order to satisfy user-supplied key-frame-like constraints • Retargeting the motion • Map the motion onto the mismatched synthetic character and modify it to satisfy important constraints • Combining the motion • Assemble motion segments into longer sequence CS, NCTU, J. H. Chuang

38. Processing the signal • Motion parameter values as a time-varying signal • Consider values of an individual parameter of the captured motion as a time-varying signal • Signal can be decomposed into frequencies, time-warped, interpolated with other signals • Multidimensional signal (vector-valued signal) • A function of time m(t): R -> Rn • Not really in Rn • 3 DOF in translation, 3 DOF in absolute orientation, many DOF in relative orientations CS, NCTU, J. H. Chuang

39. Processing the signal • Motion signal processing • Considers how frequency components capture various qualities of the motion. • Lower frequencies – represent base activities (walking) • Higher frequencies – idiosyncratic movements (walking with limp) • Motion manipulation/editing/warping • Warps the signal in order to satisfy user-supplied key-frame-like constraints. CS, NCTU, J. H. Chuang

40. Processing the signalMotion signal processing • Treat motion as a multi-dimensional signal • Low pass filtering • noise removal • High pass filtering • style change • High frequency component can be motion details, not just noise • Modify a motion through filtering is not easy • Physical constraints (joint limit, ground contact)? • Naturalness? CS, NCTU, J. H. Chuang

41. Processing the signalMotion signal processing • Frequency bands can be extracted, manipulated, and reassembled to allow the user to modify certain qualities of the signal while leaving others undisturbed. • Signal is successively convolved with expanded versions of a filter kernel. • Gains of each band are adjusted by the user and can be summed to reconstruct a motion signal. CS, NCTU, J. H. Chuang

42. Motion signal processing • PROBLEM: High frequencies can be important! • Getting rid of them makes motion look soggy • ANSWER: Do not over-apply LPF • Small amounts of Low-Pass Filtering • Noise modeling • Non-linear filters ? CS, NCTU, J. H. Chuang

43. Motion signal processing High frequencies are important! • Don’t occur often • Always significant • Impact • Rapid, sudden movement • Emphasis • Sensitivity of perception • Eye is sensitive to high frequencies CS, NCTU, J. H. Chuang

44. Motion manipulation/editing/warpingWhy? • What you get is not what you want • You get observations of the performance • Specific performer (a real human) • Specific motion • With the noise and “realism” of real sensors • You want animation • A character • Doing something • Or something similar but not the same CS, NCTU, J. H. Chuang

45. Motion manipulation/editing/warping • Manipulate time • Time warp / Speed control • m(t) = m0( f(t) ) • f : R  R • Manipulate value • m(t) = f(m0(t) ) • f : Rn Rn CS, NCTU, J. H. Chuang

46. Motion manipulation/editing/warpingManipulating time m(t) = m0( f (t) ) • Time scaling f(t) = k t • Time shifting f(t) = t + k • Time warping • Interpolate a table • Align events • Speed control • Ease in/Ease out CS, NCTU, J. H. Chuang

47. Motion manipulation/editing/warpingManipulating values • Scale • Shift • Blending • Filtering • Transition between motions • Cyclification • Change style • Constraints on the motion • Concatenation CS, NCTU, J. H. Chuang

48. Motion manipulation/editing/warpingMotion warping • Witkin and Popovic, “Motion Warping,” SIGGRAPH’95 • Keyframes as constraints in smooth deformation • Keyframe placing the ball on the racket at impact CS, NCTU, J. H. Chuang

49. Motion manipulation/editing/warpingMotion warping CS, NCTU, J. H. Chuang

50. Manipulating motion capture dataRetargeting the motion • Map the captured motion onto the mismatched synthetic character and then modify it to satisfy important constraints • Constraints: • avoid foot penetration of the floor, avoid self-penetration, no feet sliding when walking • A new motion is constructed, as close to the original motion as possible, while enforcing the constraints. • New motion is formulated as a space-time, non-linear constrained optimization problem. CS, NCTU, J. H. Chuang