Interaction with Sound

1. Interaction with Sound Explorations beyond the Frontiers of 3D virtual auditory Environments Niklas Röber, AG GAMES/UISE, ISG Otto-von-Guericke University Magdeburg, Germany

2. Motivation and Introduction Introduction– Auditory Environments – Sound Simulation – Applications – Conclusions Can one interact with a 3D environment solely using sound? • Auditory vs. visual perception • Scene sonification • Scene auralization

3. Research Goals Introduction– Auditory Environments – Sound Simulation – Applications – Conclusions • Definition, analysis and classification of 3D virtual (augmented) auditory environments • Development of … • Methods for scene and object sonification • Interaction and scene exploration techniques • A 3D audio framework • Stability of sound rendering • Quality of 3D sound spatialization • Achievable degree of realism

4. Auditory Perception Introduction– Auditory Environments – Sound Simulation – Applications – Conclusions • Environmental perception • Location and distance • 3D sound perception • Individual per person • Based on head-related transfer function (HRTF / HRIR) • Auditory perception • Auditory Gestalt [Mach86, Ehrenfels90] • Auditory scene analysis [Bregman90] • Selective listening[Williams92] 3D Sound Perception [Wikimedia]

5. Research Overview Introduction– Auditory Environments – Sound Simulation – Applications – Conclusions • 3D scene and object sonification • User input and spatial interaction techniques • Efficient sound rendering • Example applications

6. 3D virtual auditory Environments

7. Auditory Displays Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • Lack of absolute values • Lack of orthogonality • Limited spatial precision • Convey abstract information using auditory means + No screens necessary + High-temporal resolution + Rapid detection, alerting • Display classification: analogic – symbolic continuum [Kramer 94] • Auditory icons [Gaver89, Mynatt92] • Earcons, hearcons[Blattner89, Bölke95]

8. 3D auditory User Interfaces Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • Spatial interaction and gestures • Balance function with aesthetics [Vickers06] • Related work: • Auditory menu [Mynatt92, Blattner92, Brewster92, … ] • Spatial auditory display [Cohen91, Wenzel92, Walker00, Marentakis05, … ] • 3D auditory menu [Begault94, Crispien96, Kobayashi98, Frauenberger07, … ] • Main questions: • Improving 3D perception with personalized HRTFs • Task-related cataloguing of sonification techniques • Spatial sonification / interaction for virtual 3D environments

9. 3D virtual auditory Environments Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions Focusing on 3D virtual audio-onlyenvironments • Sonification (output) • Orientation, navigation, exploration, and manipulation • Binaural display • Interaction (input) • Key element: real-time interaction + sonification • Natural listening cues • Head-tracking and spatial interaction

10. Scene Auralization Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions (ICAD 2004, CGAIDE 2004) • 3D scene auralization • Sounds, speech, music • Room acoustics Not sufficient! • Non-realistic auditory scene representation • Non-physically based auralization • Additional non-object sounds • Exaggerated / suppressed parameters • Situation-based display styles (aud. landmarks, aud. texture, path sonification, menu)

11. Global Sonification/Interaction Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions (ICAD 2004, CGAIDE 2004, ICAD 2005) • Tasks: orientation, navigation, and scene exploration Passive: • Hearcons / auditory icon • Auditory landmarks • Interactables (object grouping) • Guiding systems • Detail: auditory lens • Tasks: orientation, searching • Focus on a particular direction and / or class of objects (zoom to objects on desk) • Active: • Pathway sonification (soundpipes) • Auditory lens / hear frustum

12. Local Sonification/Interaction Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions (ICAD 2004, CGAIDE 2004, ICAD 2005) • Tasks: orientation, object selection and manipulation Passive: • Hearcons / auditory icon • Detail: auditory textures • Ext. of Mynatt’s parameter nesting [Mynatt92] • Describes an object and its function using different auditory representations • Interaction dependencies (door locked, door opening, door open) • Active: • Radar / sonar • 3D pointing / selection (cane) • Auditory textures • Dependency modeling

13. Detail: Usability Analysis Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions Test environment • Usability test (14 participants) • Examination of developed sonification / interaction techniques • Qualitative evaluation • Tasks • Orientation and navigation • Find and select specific objects • Test setting • Polhemus FASTRAK (head-tracking, stylus) • Gamepad • HiFi Headphones

14. Detail: Results Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions Auditory Lens 1 – very bad 2 – bad / worse 3 – neither 4 – good / better 5 – very good Ring Menu μ: 3.79 μ: 3,57 μ: 4,00 σ: 0.8 σ: 1.0σ: 1.0 μ: 3,64 μ: 4,64 σ: 1.1 σ: 0.5 Orientation with lens Navigation with lens Assist. of head-tracking Overall interaction Item localization Gamepad vs. gestures

15. Discussion Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • 3D audio framework • Non-realistic auditory scene representation • 3D scene / object sonification • Sound rendering implemented using OpenAL Problem! OpenAL has several severe restrictions • Generalized HRTFs • Basic environmental modeling

16. Efficient Sound Rendering and Simulation

17. GPU-based Sound Signal Processing Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions Input Texture Streaming Raw Sound Data Signal Processing Filter Readback Output Texture Playback (OpenAL) • Very efficient for impulse response filtering • GPU-based sound signal processing [Whalen05, Gallo04] • General sound effects (chorus, reverb, … ) • 3D sound rendering: convolution/frequency weighting (10 bands)

18. GPU-based Room Acoustic Simulation Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions (ICMC 2006, DAFx 2007) [Smith92, VanDuyne93, Bilbao04, Campos05] • Wave propagation using time-domain difference model • Computationally very complex • Applicable to lower frequencies Wave-based Simulation • Ray/Energy-based Simulation[Funkhauser02, Tsingos04, Jedrzejewski04] • Approximates sound waves using directional rays • Applicable tomiddle and higher frequency ranges

19. Detail: Optimal Sampling Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions (ICMC 2006) • Hexagonal lattices provide a higher packing density[Conway76] • Optimal sampling: BCC lattice • Unit length increases to • Sampling efficiency: • 8 neighbors with 4 axes of propagation • Less pronounced frequency dispersion error • Update frequency with unit length changes to Cartesian lattice BCC lattice

20. Detail: GPU Implementation Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions (ICMC 2006) • Based on 3D textures, fragment shader and 3D framebuffer-objects • Shader samples texture using screen aligned slicing quads • BCC decomposed into two 3D textures • Two nodes computed in one step

21. Discussion Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • 3D waveguide meshes • Improved efficiency (20x – 60x) • Improved simulation results for BCC lattice • Ray acoustic simulation • Real-time simulation up to 30k models (incl. auralization) • Integration of wave-based effects (diffraction) • Frequency-based material modeling • Promising virtual HRIR simulations • Simulations exhibit all important features • On the way to personalized HRTFs Time (ms) Angle HRIR horizontal plane

22. Applications and Case Studies I Introduction – Auditory Environments – Sound Simulation – Applications– Conclusions (ICAD 2005, Gamesconference 2005, AudioMostly2006) • Audio-only computer games Development of games that are played solely through listening. • Three action, one auditory adventure game • Usability test • Augmented audio reality (AAR) Enhancing a real environment with additional auditory information. • Self developed AAR system • Usability test Audiogame Mosquito AAR Game The hidden Secret

23. Applications and Case Studies II Introduction – Auditory Environments – Sound Simulation – Applications– Conclusions (CGAIDE 2004, TIDSE 2006, AudioMostly 2007) • Interactive audiobooks Combining audiobooks with interactive elements from computer games. • Non-linear story graph with variable degree of interaction • Usability tests • Scene authoring environment • Extension of audio framework • Authoring of 3D sound sources, acoustics, auditory textures and ring menu systems Storytree The Pit and the Pendulum Authoring Environment

24. Summary and Contribution Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • 3D virtual auditory environments (ICAD 04, CGAIDE 04, ICAD 05, AM 08) • Non-realistic auditory scene representation • 3D scene and object sonification / interaction • 3D audio framework • Graphics-based sound rendering (ICMC 06, DAFx 07) • 3D waveguide meshes with optimal sampling • Ray acoustics simulation with diffraction and material modeling • Virtual HRIR simulations • Applications and case studies(DIGRA 05, TIDSE 06, AM 06, AM 07) • Audiogames and interactive audiobooks • Augmented audio reality

25. Conclusions Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • 3D auditory environments are as effective as visual environments • Spatial sonification / interaction • High-quality sound rendering • Audio-centered design • Domains • Audiogames and interactive audiobooks • Augmented audio reality • Aiding the visually impaired

26. Future Work Introduction – Auditory Environments – Sound Simulation – Applications – Conclusions • User interface design • Multi-user presentation and interaction • Advanced gestures • Perceptual presentation • Acoustic rendering • Comparison with real-world measurements (Bell Labs Box) • Experimentation with virtual HRIR measurements • Augmented audio reality • Improvement of positioning accuracy / system latency

27. Thank you! Questions? http://x3t.net/thesis.html A Thank You to everyone who helped and participated in this research, especially to all my students!

28. Promotionsfeier Appendix Beginn 18 Uhr Kleiner Saal der Festung Mark (Eingang Jakobstrasse, eine Treppe hoch)