Chapter 6 Audiovisual Speech Perception

1. Chapter 6Audiovisual Speech Perception Perry C. Hanavan, Au.D.

2. Questions Lipreading and speechreading difined the same in the text book? True False

3. Definitions • Lipreading • the person relies ONLY on the visual speech signal and other visual cues provided by the talker • Speechreading • the person attends to both the visual AND auditory cues provided by the talker

4. General Tendencies • Normal hearing persons rely on speechreading • Persons listening in background noise tend to use visual cues for speech recognition • Persons with hearing loss tend to rely more on the visual signal for speech recognition • The greater the hearing loss, the more the tendency for reliance on visual cues

5. Speechreading • Lip cues • Facial expression cues • Gesture cues • Body language cues • Linguistic • Situational cues • Auditory cues*

6. Speechreading for Communication • Normal hearing adults • Infants

7. Afternoon Clinic Appointments • You arrive at work following lunch and find two patients waiting for clinical services. • Dr. White is a 50 year old physician • Mr. Black is a 20 year old assembly line worker in a noisy factory • Who do you predict will have the better speechreading skills?

8. Characteristics of Good Speechreaders • Visual word decoding • Working memory (ability to store and manipulate items in memory simultaneously) • Lexical identification speed (determining whether letters are a word) • Phonological processing speed (whether two words rhyme) • Verbal inference (how well one can complete a sentence with missing words) • Age (Honnel et al 1991) (younger better)

9. Characteristics of Good Speechreaders • Ability to capitalize on contextual cues • Willingness to guess • Mental agility • Willingness to revise interpretations of partially recognized messages • Linguistic skills • World knowledge • Modal differences (Erber, 1974) • Neurophysiological (Summerfield, 1992) • Cognitive (Ronnberg et al, 1999)

10. Factors that Influence Speechreading Difficulty

11. Predictive Powers

12. Lipreading: Follow the Eyes • Lip shapes vary based on phoneme(s) produced • Eyebrows rise with questions • Tend to gaze at eyes, nose and mouth • Occasional looks at forehead, cheeks and chin • Prosodic information—lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

13. Gaze Saccades/Patterns

14. Gaze-direction-based MEG averaging during audiovisual speech perception

15. Lipreading: Follow the Eyes • Prosodic judgments: • lipreader monitored upper face such as forehead wrinkling, eyebrow rising, and eye widening

16. Lipreading: Follow the Eyes • Phonetic judgments: • monitored lower face such as lip and jaw movement

17. METS Talker Message Environment Speechreader Facial expression length viewing angle lipreading skill Diction syntactic complex distance residual hearing Body language freq of word use lighting amplification Speech rate homophenes room acoustics stress profile Speech intensity context distractions attentiveness Familiarity fatigue to lipreader motivation Facial characteristics language skills Speech prosody Objects in or over mouth

18. Difficulty Lipreading • One third speech sounds visible • mid and back consonants invisible • Vowels not highly visible • Rapidity of speech – 150 to 250 word/min • Coarticulation • Stress can change appearance of word • Talker variability • Visemes and homophenes

19. Question Define viseme: • Lip gloss • Seeing eye dog • Groups of speech sounds that APPEAR identical on the lips • Words that LOOK identical on the mouth E. C & D

20. Visemes and Homophenes • Visemes • Groups of speech sounds that APPEAR identical on the lips • p b m – f v – s z – n t d s z – k g • Homophenes 40-60 % of words • Words that LOOK identical on the mouth • pan ban man

21. Consonants Grouped as Visemes

22. Homophenous Word Pairs

23. Vowels and Lipreading • Vowels not considered highly visible • Vowels tend to be audible • Intense, long duration • Front vowels • Lips flat or spread • Back vowels • Lips rounded

24. Production of Vowels • Cardinal Vowel Chart (YouTube)

25. Consonants • Place of production • Individuals with high frequency hearing loss may have difficulty hearing place cues • F2 transition (1000-2400 Hz range) • Place cues tend to be visible • Bilabial, labiodental, linguadental sounds visible • Manner of production • Cues not visible, must be heard • F1 transition (250-1000 Hz range) • Voicing • Cues not visible • Low frequency range

26. Mumble Looks away when speaking Chews gum Unusual accent Speech impediment Smiles too much Moves around while talking No facial expression Shouts High pitch voice Talks to rapidly Uses long complicated sentences Wears a beard/mustache Wears dark glasses Difficult Speaking Behaviors

27. Talkers • Easier to lipread someone familiar • Family members, teachers, etc. • Females easier to lipread than males • However, auditory plus vision may be more difficult as females are less audible to person with hearing loss

28. Question Clear speech…which is not a principle of? • Talk slightly faster • Talk slightly louder • Talk slightly slower • Talk with some pauses • None of the above

29. Talker Clear Speech Principles • Use clear speech versus conversational speech • Slightly slower talker rate • Slightly more intensity • Slightly more pauses after key words and phrases

30. Message • Structure – complexity of message, frequency of use, linguistic context • Frequency of usage – how often a word occurs in everyday conversations • Neighborhoods – fewer lexical neighbors can be beneficial • Context – words specified by context are easier

31. Lexical Neighborhoods

32. Environment • Viewing angle – face to face • Distance – favorable seating • Room conditions – lighting, lighting angle, shining light, interfering objects, room noise

33. Speechreader • Audibility • Use of appropriate amplification system, ALDS, cochlear implant • Use of eyeglasses if necessary • Emotional and physical state

34. Tadoma Method: Speechreading • Deafblind Users of Tadoma Speechreading Method

36. Heather Whitestone: 1994 I find lip reading very stressful and frustrating because I am often confused. For example, if you look at person’s lips saying dog and saw, they look the same. With my hearing aid alone, I do not hear "s" or "d" sounds. So usually I have to use my common sense. For example, if someone said, "The dog is running across the street." Then I knew it was not the saw who ran across the street – it was the dog. Most hearing people do not understand that people in my position have to think incredibly fast in order to keep up with conversations. One-on-one conversations are not that stressful, but group conversations when coupled with background noise are nearly impossible. Lip reading is a grueling and exhausting mental exercise and lip readers are constantly thinking and trying to discern what is actually being said. I get real mad at those who think that I am stupid simply because I cannot hear. The truth is I get exhausted after a while and simply cannot keep up. At that point, I begin to guess at what is being said and eventually give up and choose to be quiet.

37. Baldi • Meet Baldi (iPhone app) • Baldi (2 iPhone app)

38. Question An oral interpreter? • Repeats message in view of person with hearing loss • Reads famous speeches • Speaks for person who uses ASL • All the above • None of the above

39. Oral Interpreters • A trained professional who sits in clear view of a person with a hearing loss and silently repeats a talker’s message as it is spoken

40. Bisensory Perceptions • What we see may influence what we hear • What we hear may influence what we see • McGurk Effect • McGurk Effect