do you see what i hear visual displays for providing the deaf with aural awareness l.
Download
Skip this Video
Download Presentation
Do you see what I hear? Visual displays for providing the deaf with aural awareness

Loading in 2 Seconds...

play fullscreen
1 / 31

Do you see what I hear? Visual displays for providing the deaf with aural awareness - PowerPoint PPT Presentation


  • 125 Views
  • Uploaded on

Do you see what I hear? Visual displays for providing the deaf with aural awareness. F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay CHI 2003 April 6 th 2003 Fort Lauderdale, FL. Outline. Background Fieldwork Two displays Experiment Field study. Background.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Do you see what I hear? Visual displays for providing the deaf with aural awareness' - Mia_John


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
do you see what i hear visual displays for providing the deaf with aural awareness

Do you see what I hear? Visual displays for providing the deaf with aural awareness

F. Wai-ling Ho-Ching, Jennifer Mankoff, James A. Landay

CHI 2003

April 6th 2003

Fort Lauderdale, FL

outline
Outline
  • Background
  • Fieldwork
  • Two displays
  • Experiment
  • Field study
background
Background
  • We experience many non-speech sounds in everyday life
    • a waterfall, a baby’s cry, music
  • In 1997 there were 3.4 million Americans with difficulty hearing who could not sense these sounds
  • Although some research has focused on language translation [Starner 97, Elliot 2000], little has explored non-speech audio awareness
non speech audio
Non-Speech Audio
  • The deaf use a variety of techniques to gain awareness of non-speech audio
    • Hearing dogs
    • Appliances fitted with flashing lights / vibration
      • doorbells, vibrating alarm clocks
    • Visual / tactile inspection
      • vibrations through the floor
  • Even with these techniques there are difficulties
the office
The Office

We saw audio awareness issues first hand while visiting the office of a deaf graduate student.

This picture of a work station is what he sees most of the day.

problem awareness
Problem: Awareness

Is my kettle done boiling?

He can not easily sense things outside his field of vision

Is my officemate at her desk?

Is someone at the door?

problem awareness7
Problem: Awareness

After interviews with deaf adults, ASL interpreters and a disability technology consultant, we uncovered general awareness issues for the deaf.

Sound based

appliances

Presence and

awareness of others

Control over

infrastructure

solution design
Solution Design

Goal: to open awareness into the wider world

  • Visual solution
    • Greater bandwidth than tactile solutions
  • Office setting
    • Greater need for assistance
    • Smaller set of sounds
    • Visual focus in one place
prototype 1 spectrograph
Prototype 1: Spectrograph

Visualization frequently used

by speech therapists

  • Color = Intensity
  • Height = Frequency
  • X-axis = Time
  • Digital sounds produce distinctive signatures
prototype 2 rings
Prototype 2: Rings

Wanted to design for novices

  • Exercise with eight hearing students to draw sound
    • 65% included the sound source
    • 59% included pictures of sound as waves or rings

Visualization includes rings & position

prototype 2 rings11
Prototype 2: Rings
  • Size of rings = Amplitude
  • Color = Amplitude
  • Center of ring = Position in a room (localization)
implementation
Implementation
  • Development Tools:
    • Python – scripting language
    • Tkinter – UI toolkit
    • SNACK – sound toolkit
    • Microphone & standard computer
    • Amplitude and pitch are

detected, but sound localization was not

implemented

experiment
Experiment
  • Dual attention signal detection task
  • Primary task
    • search in a number field
  • Secondary task
    • visual detection of target sounds
  • Trials of random length between 5-60 seconds
experiment14
Experiment
  • 8 deaf adult participants
  • 2x2x2 Repeated measures within subjects design
    • Spectrograph & rings prototypes
    • Noisy & quiet conditions
    • Knocking & phone ringing
  • Randomized order of conditions
  • 4 repetitions per condition
video
Video
  • Each of the 8 conditions is shown twice.
results
Results
  • Measurements
    • Preference – Qualitative results from survey
    • Distraction – Degradation of performance in primary task
    • Detection – Number of target sounds identified
  • Results
    • All participants preferred the Rings visualization
    • No significant distraction with either system compared to a control
results detection
Results: Detection
  • Overall detection, Rings 83%, Spectrograph 69%
  • Simple average suggests that Rings supports significantly higher detection rates
results detection18
Results: Detection
  • Closer inspection reveals more than simple averages
    • Rings better in noisy conditions
    • Spectrograph better in quiet conditions
  • Why? spectrograph good at gaining attention, poor for detection
field study back to the office
Field Study: Back to the office
  • Experiment allowed fine grained control over multiple factors, inference of causation
  • Field study allows observations of in context use, recommendations for construction of a practical system
field study
Field Study
  • Deployed Spectrograph on laptop with deaf participant for one week
  • Findings:
    • Distant sounds more difficult to detect with microphone
    • Paired training developed

“… [my officemates] liked watching the patterns of the sounds. So we were testing the display by knocking on the office door… speaking, setting the mobile phone to ring. The display performed excellently on these counts – it could show the sounds.”

-Deaf participant

discovery
Discovery

Sounds detected during field study

  • speech, mobile phone calls, door knocking, whistling kettle, chair movement, typing, mouse movement, page turning, papers rustling, footsteps and a truck outside

“This is great! … I’m learning to hear again after 30 years!”

-Deaf participant

summary
Summary
  • Designed 2 displays for the deaf based on field work
    • One based on position
    • One based on a spectrograph
  • Controlled experiment suggests
    • no significant distraction
    • results mixed as to which display is superior
  • Field study showed the spectrograph could support discovery of new sounds
do you see what i hear visual displays for providing the deaf with aural awareness23
Do you see what I hear? Visual displays for providing the deaf with aural awareness
  • Thanks go to Scott Luebking, Anthony Tang, Jeffery T. Holman, Lyn Bartram, Rashmi Sinha, Juan Carlos Lopez, Nick Chen, Gary Hsieh, Alan Liu, Mira Sutijono, our hearing and deaf participants.
  • The Group for User Interface Research, UC Berkeley Assistive Technology class.

For more information:

guir.berkeley.edu\ic2hear

user comments
User Comments

“This is great! … I’m learning to hear again after 30 years!”

“If I see a truck outside and see the display show a sound pattern that tracks the truck’s movement closely, I infer that there must be a linkage”

“When I first set up the display, I had to explain how it worked to my fellow officemates in the other room. They were fascinated and wanted to learn more about the sounds that could be detected – they liked watching the patterns of the sounds. So we were testing the display by knocking on the office door… speaking, setting the mobile phone to ring. The display performed excellently on these counts – it could show the sounds.”

state of the art
State of the Art
  • The deaf use a variety of techniques to remain aware of sound
    • Hearing dogs
    • Sensing vibrations
      • Alarm clocks, floors
    • Appliances fitted with flashing lights
      • TTY’s, doorbells
    • Touch
      • The kettle
    • Visual inspection
video27
Video
  • 2 target sounds: Cellular phone ring, Door knock, Speech
results identification
Results: Identification
  • Overall, Rings 89%, Spectrograph 70%
  • Rings better in noisy conditions
  • Spectrograph better in quiet conditions
  • Observation: spectrograph good at gaining attention, poor for identification
video29
Video
  • Each of the 8 conditions are shown twice.
do you see what i hear visual displays for providing the deaf with aural awareness30

Do you see what I hear? Visual displays for providing the deaf with aural awareness

guir.berkeley.edu\ic2hear

wai-ling@cs.berkeley.edu

jmankoff@cs.berkeley.edu

landay@cs.berkeley.edu

problem awareness31
Problem: Awareness

Is my kettle done boiling?

Sound based

appliances

Presence and awareness of others

Is my officemate at her desk?

Control over Infrastructure

Is someone at the door?

After interviews with deaf adults, ASL interpreters and a disability technology

consultant, we uncovered general awareness issues for the deaf.