Wii3D – Extending the Wiimote into 3D

1. Final Honours Presentation Wii3D – Extending the Wiimote into 3D Principal Investigator: João Lourenço Supervisor: Dr Hannah Thinyane

2. Presentation Outline • Introduction • Design and Implementation • User Study • Future Work • Conclusion

3. Introduction • Problem Statement • Research Motivation and Goals

4. Introduction • Problem Statement • Research Motivation and Goals

5. Problem Statement • Migration from 2D output to 3D output • 3D TV • Surround sound speakers • Suggests a need for migration from 2D input to 3D input • Microsoft Kinect (previously Project Natal) • Wii3D

6. Introduction • Problem Statement • Research Motivation and Goals

7. Research Motivation and Goals • Intuitive 3D Interaction Techniques are needed • Develop 3D Interaction using Nintendo Wii Remote • Compare Proposed interaction technique with existing techniques

8. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

9. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

10. Hardware • 2 Nintendo Wii Remotes • Personal Computer with Bluetooth Capabilities • 2 WiiGloves

11. Nintendo Wii Remotes • Layout Maximizing Volume

12. Nintendo Wii Remotes • Layout Maximizing Efficiency

13. WiiGlove • Circuit • 2 Configurations • 2 IRs per hand • 4 IRs on one hand

14. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

15. OS and Frameworks • Windows 7 • Microsoft C#.NET 4.0

16. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

17. Bluetooth • Microsoft Windows built-in Bluetooth Stack

18. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

19. Libraries • WiimoteLib 1.7.0.0 – Wiimote Connection • Accord.NET 2.1.1.0 – Statistics • Math.NET Iridium 2008.8.16.470 – Linear algebra • 32feet.NET 2.2.0.0 – Bluetooth Connection

20. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

21. Point Tracker • Point tracking and probabilistic estimation were required to ensure that the system would handle multiple inputs (multitouch) and continue to function even in the event of the loss of points. A loss of points can occur for several reasons: • An infra-red source can leave the field of view of the Wii Remote • An infra-red source can move behind another infra-red source • Two infra-red sources can come together and merge

22. Point Tracker • Several Components • Stereoscopic Triangulation Solver • Distance Minimizer • Predictive Interpolator

23. Stereoscopic Triangulation • Achieved by solving for the point at the near intersection of two rays • Camera intrinsic properties needed • Viewport width • Viewport height • Horizontal FOV • Vertical FOV

24. Stereoscopic Triangulation

25. Stereoscopic Triangulation • Using the fields of view, a vector from the camera position to the detected point was obtained. These vectors can be thought of as rays from the infra-red source to the camera. • σthe horizontal field of view • ψbe the vertical field of view • x be the horizontal component of the two dimensional coordinate • y be the vertical component of the two dimensional coordinate

26. Stereoscopic Triangulation

27. Stereoscopic Triangulation • With the two rays from the two Wii Remote Cameras, the intersection of the two rays can be calculated using linear algebra.

28. Stereoscopic Triangulation

29. Stereoscopic Triangulation • Physical System – rays would never perfectly intersect • Need to take an estimate • Estimate distance along first vector • Estimate distance along second vector • Average the two points and make note of the distance between them

32. Distance Minimization • The points that are reported by the two Wii Remotes are not necessarily sorted in the same order

33. Distance Minimization • Pairwise permutation of points in matrix form • Minimize distance error per row

34. Predictive Interpolation • Try to predict where a point should be – expected points • Instantaneous velocity • Average velocity over a period • Minimize the errors between the observed and expected points

35. Predictive Interpolation

36. Predictive Interpolation • The selected point per row is used as the point that the Wii3D System reports

37. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

38. Gestures and Gesture Recognition • Supported gestures • Recognition methods

39. Click Gesture • A finger going down and forward, and then up and backward

40. Pan/Scroll Gesture • Two fingers that are close together moving in similar directions

41. Zoom Gesture • Two fingers moving in opposite directions with the distance between them increasing

42. Rotate Gesture • Two fingers moving in opposite directions with the distance between them constant

43. Circle Gesture • Hidden Markov Model Gesture

44. Multitouch Gestures • Number of points used for each gesture has been linked to the intensity of the gesture

45. Recognition • Movement – polling • Simple gestures – Finite State Automata • Complex Gestures – Hidden Markov Models

46. Design and Implementation • Hardware • Software • OS and Frameworks • Bluetooth • Libraries • Point Tracker • Gestures and Gesture Recognition • User Interface

47. User Interface • States:

48. User Interface • Upon start-up, the Wii3D System attempts to connect to the Wii Remotes

49. User Interface • Two possible errors – not 2 Wiimotes connected or no Wiimotes found

50. User Interface • Search for devices