Spatialized Audio in the Common Operating Perspective

1. Spatialized Audio in the Common Operating Perspective Ryan Kilgore Mark Chignell University of Toronto Interactive Media Lab Capt Stephen Boyne Nada Pavlovic Defence Research and Development Canada NATO RTO VizCOP 14-17 Sep 04 Toronto, ON

2. Overview • Motivate use of Spatial Audio in COP • Human Factors Issues and Sample Uses • Introduction to spatial audio • The Vocal Village: A spatialized VoIP system • Experimental Results • Proposals for Future Use

3. “Common Operating Perspective”? • The perspective is not a picture, it is a mental model built in the head of the commander/operator based on information inputs • “Picture” ignores the role of audio and other senses. Humans are multi-modal creatures. • “COP is a shared mental image of what is going on” • Audio should be part of the perspective

4. Requirements • Support close interaction of a “mixed expert team” viewing the same picture • Give the involved people “close to real-time” situation awareness • Be intuitively understandable without special training • Minimize the need for mental transformations and cognitive effort • Available now or in near future • “let’s try stuff out and see what works” • 70% working solution is better than 110% unavailable

5. Human Factors Issues • Stress and Performance • Workload and Time stress • Attentional Resource Theory • Situation Awareness, Monitoring • Alerting and Orienting • Discrimination of signals in noise

6. Stress and Performance

7. Multiple Attentional Resources

8. Audio Applications for COP • Cueing for multiple screen displays • Monitoring multiple audio streams simultaneously • Different audio info dependent upon listener’s location in CP • Development of an always on audio space with spatialized components

9. Usage Considerations • Allow personalization of audio space • Avoid mental transformations • Locate different auditory displays and communication channels in different spatial positions • Distributed Speaker arrays can be used to provide required spatial audio effects

10. spatial audio | explanation • Similar to binocular vision • Result of the brain’s ability to perceive relative differences between signals picked up by the left and right ears (time, volume, frequency) • Allows people with binaural hearing to locate sound sources in three-dimensional space Spatialized audio: worth considering for COP?

11. spatial audio | benefits • Laboratory experimentation and realistic scenarios (e.g., use by pilots) have demonstrated numerous cognitive benefits of spatial audio when compared to non-spatial audio, including: • increased intelligibility of speech • improved detection of signals at lower signal-to-noise ratios • improving memory, comprehension, and speaker identification in audioconference environments Spatialized audio: worth considering for COP?

12. the Vocal Village | overview • Real-time VoIP audioconferencing through client/server architecture • Low-fidelity binaural cues allow for the presentation of individual conferee voices from different apparent locations, low bandwidth requirements • Voice locations may be automatically controlled or manually customized by individual participants through a Graphical User Interface • Minimal requirements placed on client for maximum portability Spatialized audio: worth considering for COP?

13. experiment | design • Subjects listened to a series of four pre-recorded audioconferences held between the same four women. The conferences were presented in four different formats, using a within-subjects design: • Nonspatialized audio (Mono) • Low-fidelity spatialized audio, with conferees in random positions (Random) • Low-fidelity spatialized audio, with conferee location determined by subject (Vocal Village) • Audio spatialized using commercial CoolEdit 3D software, with speakers in random positions (CoolEdit) Spatialized audio: worth considering for COP?

14. experiment | results User Preference: Audio format had a significant effect on User Preference, with the personalized, spatial Vocal Village format being the most preferred (1 is “best” in the table below). F[3,66] = 18.77, p < 0.001 Spatialized audio: worth considering for COP?

15. experiment |results • Perceived Attention Allocation: • Varied significantly by audio format (F[3,66] = 6.572, p = 0.001). • Personalized spatial display best • Spatialized conference better than mono Spatialized audio: worth considering for COP?

16. experiment |results Perceived Speaker Identification Difficulty: Easier for spatialized conference F[3,66] = 7.44, p < 0.001 Spatialized audio: worth considering for COP?

17. experiment |conclusions • Low-fidelity, “within-the-head” spatialization techniques implemented within the Vocal Village tended to increase objective performance (not significant) • Personalized spatialization provided by the Vocal Village significantly improves user perceptions of performance (increased confidence and ease of speaker identification) • The Vocal Village environment was significantly preferred by users to traditional audioconferencing Spatialized audio: worth considering for COP?

18. spatialized audio and situation awareness • Study looked at spatialized radio messages in MOUT environment • 3 Levels of fidelity examined • Stereo • Generic HRTF • Free Field

19. Results • Spatial Audio improved message location identification • Lack of head tracking appears to diminish effect of spatialized audio

20. Proposal/Conclusions • Technology for presenting spatial audio in the COP is available • Picking up the phone to communicate should be replaced with sophisticated audio spaces that promote awareness and offload visual processing • Spatial Audio can also be used to simplify the task of orienting to alerts and messages • To try out spatialized audioconferencing download the client from: • www.vocalvillage.net (chignell@mie.utoronto.ca) for more info on the Vocal Village

21. Spatialized audio in situation awareness • Study to examine use of spatialized audio cues in target localization and memory for spatial layout • In support of 3D sensor views and 2D topographic displays • Different audio configurations may aid different tasks