The Impact of the Combination of Real Object Interaction and Visual Overlay on Learning

1. The Impact of the Combination of Real Object Interaction and Visual Overlay on Learning By John Quarles

2. Presentation Outline • Introduction • Previous Work in Mixed Reality • Anesthesia Machine Training • My Previous Work • Research Plan • Timeline

3. Introduction • Mixed Reality (MR) combines the real world with a virtual world. • MR is used in many training applications • Example: Flight Simulator • Can MR technology effectively improve learning? • MR technology needs to be evaluated

4. Introduction • Fused Worlds (FWs) • Visual Overlay (i.e. 3D graphics) • Real and virtual object interaction How do FWs impact learning?

5. Introduction • 3 interdependent aspects of learning: • Learning Efficiency • Rate of learning • Learning Quality • Mental models • Learning Flexibility • Generalization

6. Introduction Fused Worlds Learning Efficiency Real and virtual object visualization Quality Real and virtual object interaction Flexibility How do FWs impact learning? How do aspects of FWs impact learning?

7. Introduction • My Previous Work • Studies comparing Real + Virtual Object Interaction to Virtual Object interaction • Real object interaction improved task learning

8. Introduction • Proposed Work • Real world app: Anesthesia Machine Training • Focused on learning • Can benefit from FWs • Plan: • Augment an anesthesia machine with a FW • Observe students as learning with the augmented machine • Framework: Combining Simulation and Realilty

9. Introduction • Broader Impact • Establish the efficacy of FWs for learning. • Show where MR tech needs improvement. • Contribute to literature on MR, simulation, visualization and HCI. • Directly affects Anesthesiologists and others in related fields

10. Presentation Outline • Introduction • Previous Work in Mixed Reality • Anesthesia Machine Training • My Previous Work • Research Plan • Timeline

11. Previous Work in MR • Mixed Reality: • Combines real world with virtual world

12. Previous Work in MR • Registration • Tracking • Displays • Interfaces

13. Previous Work in MR • Displays • Magic Lenses • Originated as 2D GUI manipulators • Later Implemented as a 3D TUI

14. Future Work in MR • Evaluation • How does MR impact the user? • Real world applications • Common Frameworks • Portability • Code Reuse • Authoring

15. Presentation Outline • Introduction • Previous Work in Mixed Reality • Anesthesia Machine Training • My Previous Work • Research Plan • Timeline

16. The Virtual Anesthesia Machine (VAM) • Simulation of a generic Anesthesia Machine • Interactive • Web disseminated

17. The VAM • Interaction with the VAM • Users interact with animated icons that represent components of the real machine.

18. Learning Tools • The VAM • The Opaque VAM • The Augmented Anesthesia Machine • Real Anesthesia Machine

19. Presentation Outline • Introduction • Previous Work in Mixed Reality • Anesthesia Machine Training • My Previous Work • Research Plan • Timeline

20. My Previous Work • Investigated the impact of FWs on learning an abstract task in a lab environment. • Specifically: Benefits of Real Object Interaction

21. My Previous Work • Task Description • 3D spatial task • Metrics • Errors • Time

22. My Previous Work • Overall Results • When interacting with real objects participants: • made fewer errors (0.33 errors/min vs 2.04 errors/min) • completed the task faster (108 sec vs 133 sec). • Real object interaction improved participants’ learning quality and learning efficiency.

23. Presentation Outline • Introduction • Previous Work in Mixed Reality • Anesthesia Machine Training • My Previous Work • Research Plan • Timeline

24. Research Plan • Thesis Statement: • MR’s combination of overlaid virtual objects and real object interaction (fused worlds) will significantly improve user learning efficiency, quality and flexibility more than overlaid virtual objects or real objects alone. • This contextualization (the FW) of abstract simulation and that which is being simulated improves overall user learning of the application.

25. Research Plan • Application: Anesthesia Machine Training • Research Overview • Augment an anesthesia machine with a FW • Compare student’s learning with and without the FW • Create a Framework that aids in combining simulation with reality

26. Research Plan • Augmented Anesthesia Machine (AAM) Fusing Worlds • Fuses the VAM and the real machine

27. Research Plan • AAM Visualization • Superimpose the VAM simulation over the real machine. • Use Computer Vision based tracking systems

28. Research Plan • AAM Interaction • Interact naturally with the real machine

29. Research Plan • Study: Discover if and how FWs impact learning. • Investigate two main aspects of FWs: • Virtual objects shown in context with the real world • Real object interaction • Compare students who learn using a FW (the AAM) to students who learn without the FW

30. Study Procedure Training Exercises 1st year Anesthesiology Residents AAM Real Machine 90 minlecture Hands-on test Pretest Posttest 3D Viz VAM 24 Hours

31. Research Plan • Metrics • Multiple choice pre/post tests • Questions come from review materials for the American Board of Anesthesiology Exam • Hands-on test • A fault is induced in the real machine and students must find and correct it. • Number of lessons completed

32. Research Plan • Many other potential applications • VAM-like simulations prevalent in training • Code/Modeling Framework • Library to help users • define the mapping between simulation and reality • visualize the mapping • define and model interaction methods

33. Presentation Outline • Introduction • Previous Work in Mixed Reality • The Virtual Anesthesia Machine • My Previous Work • Research Plan • Timeline

34. Timeline • Previous Publications • ISMAR, 3DUI • Anticipated Publications: • Simulation: TOMACS, SIMULATION • MR/HCI: ISMAR, VR, CHI • Visualization: IEEE Viz • Med/Psych Journals

35. Timeline • Workflow

36. Acknowledgements • Committee Chair: Ben Lok • Committee Members: • Ira Fischler • Paul Fishwick • Jeff Ho • Sem Lampotang

37. Fin.

38. Detailed Timeline • VR’08, TVCG Journal “The Impact of the Augmented Anesthesia Machine on Learning about Anesthesia Machines” • VR’09 or ISMAR’08: “How Real Object Interaction Impacts Learning in Anesthesia Machine Training” • CHI’09:“How Visualization Superimposed Over the Real World Impacts Learning in Anesthesia Machine Training” • CHI Journal or PresenceJournal: “The Impact of Fused Worlds on Learning about Anesthesia Machines” • Medical/Psychology Journals.

39. Detailed Timeline • Anticipated Publications • TOMACS: "An Approach to Contextualizing Simulation Models with Physical Phenomena with an Application to Anesthesia Machines" • ISMAR’07: “Using Mixed Reality to Effectively Combine Real Object Interaction and Abstract Simulation” • ISMAR’07: “The Impact of Real Object Interaction on Spatial Memory and Learning”

40. Summary • Purpose: To discover if and how FWs impact learning efficiency, quality, and flexibility? • Build the AAM and Framework • Run a major study • Combined aspects of FWs are greater than individual aspects?

41. Summary • Completing this work will: • Establish the efficacy of MR for learning efficiency, flexibility, and quality objectives • Highlight the areas of MR technology that need to be improved upon in future MR research. • Contribute to the literature on haptics, and MR interaction.

42. Introduction • Broader Impact • Benefits VR/MR community • Researchers • App Developers • Benefits Anesthesiology community • Doctors • Students

43. Summary • Innovations • Creating a way to combine 2D abstract simulation with the real world • Determining how FWs impact user learning

44. Research Plan • AAM Visualization Method 2 • Superimpose the real machine over the VAM.