Do you see what I hear? Visual displays for providing the deaf with aural awareness

Do you see what I hear? Visual displays for providing the deaf with aural awareness F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay CHI 2003 April 6th 2003 Fort Lauderdale, FL

Outline • Background • Fieldwork • Two displays • Experiment • Field study

Background • We experience many non-speech sounds in everyday life • a waterfall, a baby’s cry, music • In 1997 there were 3.4 million Americans with difficulty hearing who could not sense these sounds • Although some research has focused on language translation [Starner 97, Elliot 2000], little has explored non-speech audio awareness

Non-Speech Audio • The deaf use a variety of techniques to gain awareness of non-speech audio • Hearing dogs • Appliances fitted with flashing lights / vibration • doorbells, vibrating alarm clocks • Visual / tactile inspection • vibrations through the floor • Even with these techniques there are difficulties

The Office We saw audio awareness issues first hand while visiting the office of a deaf graduate student. This picture of a work station is what he sees most of the day.

Problem: Awareness Is my kettle done boiling? He can not easily sense things outside his field of vision Is my officemate at her desk? Is someone at the door?

Problem: Awareness After interviews with deaf adults, ASL interpreters and a disability technology consultant, we uncovered general awareness issues for the deaf. Sound based appliances Presence and awareness of others Control over infrastructure

Solution Design Goal: to open awareness into the wider world • Visual solution • Greater bandwidth than tactile solutions • Office setting • Greater need for assistance • Smaller set of sounds • Visual focus in one place

Prototype 1: Spectrograph Visualization frequently used by speech therapists • Color = Intensity • Height = Frequency • X-axis = Time • Digital sounds produce distinctive signatures

Prototype 2: Rings Wanted to design for novices • Exercise with eight hearing students to draw sound • 65% included the sound source • 59% included pictures of sound as waves or rings Visualization includes rings & position

Prototype 2: Rings • Size of rings = Amplitude • Color = Amplitude • Center of ring = Position in a room (localization)

Implementation • Development Tools: • Python – scripting language • Tkinter – UI toolkit • SNACK – sound toolkit • Microphone & standard computer • Amplitude and pitch are detected, but sound localization was not implemented

Experiment • Dual attention signal detection task • Primary task • search in a number field • Secondary task • visual detection of target sounds • Trials of random length between 5-60 seconds

Experiment • 8 deaf adult participants • 2x2x2 Repeated measures within subjects design • Spectrograph & rings prototypes • Noisy & quiet conditions • Knocking & phone ringing • Randomized order of conditions • 4 repetitions per condition

Video • Each of the 8 conditions is shown twice.

Results • Measurements • Preference – Qualitative results from survey • Distraction – Degradation of performance in primary task • Detection – Number of target sounds identified • Results • All participants preferred the Rings visualization • No significant distraction with either system compared to a control

Results: Detection • Overall detection, Rings 83%, Spectrograph 69% • Simple average suggests that Rings supports significantly higher detection rates

Results: Detection • Closer inspection reveals more than simple averages • Rings better in noisy conditions • Spectrograph better in quiet conditions • Why? spectrograph good at gaining attention, poor for detection

Field Study: Back to the office • Experiment allowed fine grained control over multiple factors, inference of causation • Field study allows observations of in context use, recommendations for construction of a practical system

Field Study • Deployed Spectrograph on laptop with deaf participant for one week • Findings: • Distant sounds more difficult to detect with microphone • Paired training developed “… [my officemates] liked watching the patterns of the sounds. So we were testing the display by knocking on the office door… speaking, setting the mobile phone to ring. The display performed excellently on these counts – it could show the sounds.” -Deaf participant

Discovery Sounds detected during field study • speech, mobile phone calls, door knocking, whistling kettle, chair movement, typing, mouse movement, page turning, papers rustling, footsteps and a truck outside “This is great! … I’m learning to hear again after 30 years!” -Deaf participant

Summary • Designed 2 displays for the deaf based on field work • One based on position • One based on a spectrograph • Controlled experiment suggests • no significant distraction • results mixed as to which display is superior • Field study showed the spectrograph could support discovery of new sounds

Do you see what I hear? Visual displays for providing the deaf with aural awareness • Thanks go to Scott Luebking, Anthony Tang, Jeffery T. Holman, Lyn Bartram, Rashmi Sinha, Juan Carlos Lopez, Nick Chen, Gary Hsieh, Alan Liu, Mira Sutijono, our hearing and deaf participants. • The Group for User Interface Research, UC Berkeley Assistive Technology class. For more information: guir.berkeley.edu\ic2hear

User Comments “This is great! … I’m learning to hear again after 30 years!” “If I see a truck outside and see the display show a sound pattern that tracks the truck’s movement closely, I infer that there must be a linkage” “When I first set up the display, I had to explain how it worked to my fellow officemates in the other room. They were fascinated and wanted to learn more about the sounds that could be detected – they liked watching the patterns of the sounds. So we were testing the display by knocking on the office door… speaking, setting the mobile phone to ring. The display performed excellently on these counts – it could show the sounds.”

What do we hear?

State of the Art • The deaf use a variety of techniques to remain aware of sound • Hearing dogs • Sensing vibrations • Alarm clocks, floors • Appliances fitted with flashing lights • TTY’s, doorbells • Touch • The kettle • Visual inspection

Video • 2 target sounds: Cellular phone ring, Door knock, Speech

Results: Identification • Overall, Rings 89%, Spectrograph 70% • Rings better in noisy conditions • Spectrograph better in quiet conditions • Observation: spectrograph good at gaining attention, poor for identification

Video • Each of the 8 conditions are shown twice.

Do you see what I hear? Visual displays for providing the deaf with aural awareness guir.berkeley.edu\ic2hear wai-ling@cs.berkeley.edu jmankoff@cs.berkeley.edu landay@cs.berkeley.edu

Problem: Awareness Is my kettle done boiling? Sound based appliances Presence and awareness of others Is my officemate at her desk? Control over Infrastructure Is someone at the door? After interviews with deaf adults, ASL interpreters and a disability technology consultant, we uncovered general awareness issues for the deaf.

Do you see what I hear? Visual displays for providing the deaf with aural awareness