Mitsubishi Electric Research Labs (MERL) Cambridge, MA

1. Instant Replay: Inexpensive High Speed Motion Capture Mitsubishi Electric Research Labs (MERL) Cambridge, MA Hideaki Nii, Jay Summet,Yong Zhao, Jonathan Westhues, Paul H. Dietz, Shree K. Nayar, John Barnwell, Michael Noland, Vlad Branzoi, Erich Bruns, Masahiko Inami, & Ramesh Raskar

2. Demo Video

3. Optical Capture • Goal • Find 2D or 3D location of tags • Approach • Each tag has a photosensor • Label space using zones created by special high speed projectors • Find zone in which the tag lies

4. Key Features • 500 Hz Tracking • Id for each Marker Tag • Capture in Natural Environment • Visually imperceptible tags • Photosensing Tag can be hidden under clothes • Ambient lighting is ok • Unlimited Number of Tags Allowed • Base station and tags only a few 10’s $

5. Potential Applications • High Speed Motion Capture & Tracking • Motion Capture • Character Animation • CG alignment • Industrial applications • Crash Test Dummies • Robot Guidance • Indoor Positioning for Ubiquitous Computing • Locating users and devices • 3D User Input

6. Existing Solutions • Optical: Uses expensive cameras to locate tags in the environment. • Passive tags give location, but no identity, leading to reacquisition problems • Magnetic • Environmental interference, expensive • Inertial Tracking (Gyro/ Compass) • Drift, Calibration

7. Solution Overview • Use inexpensive solid state projectors to “label space”. • Solid State lighting (LED’s) can be modulated extremely quickly (MHz to GHz). • Each point in space receives a different code sequence. • Inexpensive and small tags receive the codes, and can detect their own location.

8. Labeling Space Each tag receives a unique signal based upon it’s physical location.

9. How Labeling Works Light source Screen Optics GrayCode Mask pos=0 pos=15 Light source blink one by one and each position on the screen has different light pattern. 4 light make 4 bit position resolution

10. System Hardware • One projector can send 10 bit one dimensional position data. • No limitation on the number of tags.

11. Inside of Projector Condensing Optics Light Source Focusing Optics Gray code Slide The Gray code pattern

12. Inside of Tag

13. Specification

14. Low cost Proto type system Mass Produced System • Projector • Electrical parts:$50 • Projector body:$10 • Optics: $ hundreds • (4 custom glass lenses) • Tag • Electrical parts:$50 • Wireless unit:$50 • Projector • Electrical parts:$30 • Projector body:$10 • Optics:$10 (Plastic lens) • Tag • Electrical parts:$10 • Wireless unit: Optional

15. Expand to 2D/3D System 2D measurement 3D measurement X2 data X data X data Y data Y data

16. Result Videos

17. Tags Hidden Under Costume • Tags can be hidden under an actors costume with only small sensors exposed. • This enables motion capture during primary filming.

18. High Speed Tracking

19. Flying Tags!

20. E-Tech Demo

21. Key Features • 500 Hz Tracking • Id for each Marker Tag • Capture in Natural Environment • Visually imperceptible tags • Photosensing Tag can be hidden under clothes • Ambient lighting is ok • Unlimited Number of Tags Allowed • Base station and tags only a few 10’s $

22. Take Away Points • Very Inexpensive Projectors and Tags. • Reasonable accuracy which will be improved with better optical design. • Very fast sampling rates possible! • We want to license the technology!

23. Thank you! Contact: Ramesh Raskar raskar@merl.com

24. New Smaller Tag