Using an Event-Based Approach to Improve the Multimodal Rendering of 6DOF Virtual Contact

1. Using an Event-Based Approach to Improve the Multimodal Rendering of6DOF Virtual Contact Jean Sreng, Florian Bergez, Jérémie Le Garrec, Anatole Lécuyer, Claude Andriot jean.sreng@inria.fr

2. Outline Context Contact states and events Computation Multimodal feedback Visual Audio 6DOF Haptic Conclusion OUTLINE

3. Outline • Context • Contact states and events • Computation • Multimodal feedback • Visual • Audio • 6DOF Haptic • Conclusion OUTLINE

4. Context • Virtual contact between objects • Virtual prototyping • Assembly/Maintenance simulations • Video games • …

5. 500 Hz 1 kHz 48 kHz 60Hz Context • Multimodal rendering of contact • Visual • Audio • Haptic • Some limitations • Computational resources • Sensor resolution

6. Context • To overcome these limitations : • Sound synthesis algorithms (modal synthesis) [van den doel et. al.] • Event-based haptic [kuchenbecker et. al.] • … • However : • Often tied to a specific physical engine • Demonstrated in a simple context

7. We propose… • An event-based approach : • Integrate various techniques • Visual • Audio • 6DOF Haptic • Based only on position information • Not to be tied to a specific physical engine • Can be easily adapted to an existing simulation

8. Outline • Context • Contact states and events • Computation • Multimodal feedback • Visual • Audio • 6DOF Haptic • Conclusion OUTLINE

9. Contact information (states) • Continuous, “time independent” states : • Free motion (no contact) • Friction (contact) Free motion Friction

10. Contact information (events) • Discrete events describing the “evolution” : • Impact • Detachment Free motion Friction Impact Detachment

11. Contact information (states and events) • The classic continuous information : • Is essential to graphic and closed-loop haptic rendering • Cannot convey efficiently the transient nature of an impact • The different states and events information can be used to improve the rendering of : • Transient phenomena (impact) • Continuous phenomena (friction)

12. Outline • Context • Contact states and events • Computation • Multimodal feedback • Visual • Audio • 6DOF Haptic • Conclusion OUTLINE

13. Computation of contact states and events • The contact condition : • The events are defined by : • Impact : • Detachment : Friction Impact Detachment Local linear velocity Normal

14. Velocity computation • To estimate the velocity : • Some issues : • Time stepping • Approximation errors

15. Computation of contact states and events • On the timestep • Impact event : • Friction state : • Detachment event :

16. Computation of contact states and events • In a nutshell… Friction Impact Detachment

17. Outline • Context • Contact states and events • Computation • Multimodal feedback • Visual • Audio • 6DOF Haptic • Conclusion OUTLINE

18. Multimodal feedback • The generated states and events can be used with specific rendering techniques • Implementation on an existing 6DOF haptic simulation

19. Visual feedback • Particle effect : • Impact event • Friction state • Detachment event • Pencil effect on friction state : • Color : Force • Thickness : Velocity

20. Audio feedback • Modal synthesis of impact event and friction state :

21. Haptic feedback • 6DOF event-based haptic feedback of impact event • Traditional closed loop feedback • High frequency force pattern Elementary wrench associated to an impact point

22. Haptic feedback

23. Conclusion • We proposed : • An event-based approach to improve the rendering of contact : • Impact • Friction • Detachment • A computation method based only on object’s positions • A multimodal implementation on a complex virtual prototyping simulation • Visual • Audio • 6DOF Haptic

24. Further work • A formal user evaluation • Compare with traditional feedback • Complex industrial context • Test some other feedback : • Detachment event • 6DOF haptic texture

25. Questions ? Thank you for listening Questions ?