From Sound to Sense and back again:
This presentation is the property of its rightful owner.
Sponsored Links
1 / 63

From Sound to Sense and back again: The integration of lexical and speech processes PowerPoint PPT Presentation


  • 225 Views
  • Uploaded on
  • Presentation posted in: General

From Sound to Sense and back again: The integration of lexical and speech processes. David Gow Massachusetts General Hospital. Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester. Sense. Sound. The Speech Chain.

Download Presentation

From Sound to Sense and back again: The integration of lexical and speech processes

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


From sound to sense and back again the integration of lexical and speech processes

From Sound to Sense and back again:

The integration of lexical and speech processes

David Gow

Massachusetts General Hospital

Bob McMurray

Dept. of Brain and Cognitive Sciences

University of Rochester


From sound to sense and back again the integration of lexical and speech processes

Sense

Sound

The Speech Chain

Complex computations from sound to sense must be broken up for study.

Assume intermediate representations:

Phonemes…

Words…

Syntactic Phrases…


From sound to sense and back again the integration of lexical and speech processes

The Standard Paradigm

Sense

The Standard Paradigm

Words

Phonology

Phonemes

Sound


From sound to sense and back again the integration of lexical and speech processes

Phonemes*essential

* or other sublexical category

The Standard Paradigm

Sense

The Standard Paradigm

Delimited fields of study.

  • Speech Perception

Words

  • Spoken Word Recognition

Phonology

Phonemes

  • Phonology

Sound


From sound to sense and back again the integration of lexical and speech processes

100

100

Discrimination

% /p/

  • Sharp identification of tokens on a continuum.

Discrimination

ID (%/pa/)

0

0

B

VOT

P

  • Discrimination poor within a phonetic category.

Why?

Categorical Perception (CP)

Continuous Acoustic Detail => Discrete Categories

Does CAD affect speech categorization?


From sound to sense and back again the integration of lexical and speech processes

Sense

Categorical Perception (CP)

  • Defined fundamental computational problems.

  • CP is output of

    • Speech perception

  • Input to

    • Phonology

    • Word recognition.

Words

Phonology

Phonemes

Sound


From sound to sense and back again the integration of lexical and speech processes

CP

  • But…

    • Not all speech contrasts are categorical.

    • Lots of tasks show non-categorical perception.

Fry, Abramson, Eimas & Liberman (1962) Pisoni & Tash (1974) Pisoni & Lazarus (1974) Carney, Widden & Viemeister (1977) Hary & Massaro (1982) Pisoni, Aslin, Perey & Hennessy (1982) Healy & Repp (1982) Massaro & Cohen (1983) Miller (1997) Samuel (1997)…


From sound to sense and back again the integration of lexical and speech processes

Sense

?

CP

Words

CP tasks don’t necessarily tap a stage of this problem.

Sound

Lexical activation… seems a good bet.

Why has the Standard Paradigm persisted?

Categorical Perception is

about phonetic classification.

The minimal computational problem: compute meaning from sound.


From sound to sense and back again the integration of lexical and speech processes

Why has the Standard Paradigm persisted?

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.


From sound to sense and back again the integration of lexical and speech processes

Segmentation

Cue extra-segmental process.

Why has the Standard Paradigm persisted?

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.

  • Example: Word Segmentation

    • Vowel Length

    • Stress/Meter

    • Coarticulation

Words

Phonemes

Word Recognition

CAD


From sound to sense and back again the integration of lexical and speech processes

Does continuous acoustic detail affect interpretation via core word-recognition processes?

  • No.Standard Paradigm is fine…

    • Yes.Hmm…

Sublexical Filter

(phonemes)

  • Need to use stimuli with:

    • Precise control over CAD

  • Need to use tasks that:

    • reflect only minimal computational problem: meaning.

    • are sensitive to acoustic detail.


From sound to sense and back again the integration of lexical and speech processes

Visual World Paradigm

Visual World Paradigm

  • Subjects hear spoken language and manipulate objects in a visual world.

  • Visual world includes set of objects with interesting linguistic properties (names)

  • Eye-movements to each object are monitored throughout the task.

Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995)

Allopenna, Magnuson & Tanenhaus (1998)


From sound to sense and back again the integration of lexical and speech processes

  • Meaning based, natural task: Subjects must interpret speech to perform task.

  • Fixation probability maps onto dynamics of lexical activation.

  • Context is controlled:

  • meaning  lexical activation.

    • Eye-movements fast and time-locked to speech.


    From sound to sense and back again the integration of lexical and speech processes

    ?

    Does continuous acoustic detail affect interpretation?

    Is lexical activation sensitive to continuous acoustic detail?


    From sound to sense and back again the integration of lexical and speech processes

    McMurray, Tanenhaus & Aslin (2003)

    • Combine tools of

      • speech perception:

        • 9-step VOT continuum.

    • spoken word recognition:

      • visual world paradigm


    From sound to sense and back again the integration of lexical and speech processes

    Methods

    A moment to view the items


    From sound to sense and back again the integration of lexical and speech processes

    500 ms later


    From sound to sense and back again the integration of lexical and speech processes

    Bear

    Repeat 1080 times…


    From sound to sense and back again the integration of lexical and speech processes

    200 ms

    Trials

    1

    2

    3

    4

    5

    Time

    Target =Bear

    Competitor =Pear

    Unrelated =Lamp, Ship


    From sound to sense and back again the integration of lexical and speech processes

    0.9

    0.8

    0.7

    0.6

    0.5

    0.4

    Fixation proportion

    0.3

    0.2

    0.1

    0

    0

    400

    800

    1200

    1600

    VOT=0 Response=

    Time (ms)


    From sound to sense and back again the integration of lexical and speech processes

    target

    Fixation proportion

    Fixation proportion

    time

    time

    Predictions

    What would lexical sensitivity to CAD look like?

    Systematic effect on competitor dynamics.

    Fixations to the competitor.

    Categorical Results

    Gradient Effect

    target

    target

    competitor

    competitor

    competitor

    competitor


    From sound to sense and back again the integration of lexical and speech processes

    20 ms

    25 ms

    30 ms

    10 ms

    15 ms

    35 ms

    40 ms

    0.16

    0.14

    0.12

    0.1

    0.08

    0.06

    0.04

    0.02

    0

    0

    400

    800

    1200

    1600

    0

    400

    800

    1200

    1600

    2000

    Results

    Response=

    Response=

    VOT

    VOT

    0 ms

    5 ms

    Competitor Fixations

    Time since word onset (ms)


    From sound to sense and back again the integration of lexical and speech processes

    P

    B

    Sh

    L

    Task?

    Phoneme ID

    Not part of minimal

    computational problem.

    Same stimuli in

    metalinguistic task…

    …more categorical pattern of fixations

    Continuous acoustic detail is not helpful in metalinguistic tasks…


    From sound to sense and back again the integration of lexical and speech processes

    Summary

    Word recognition shows gradient sensitivity to continuous acoustic detail.

    Not extra-segmental: VOT

    CAD affects higher-level processes.

    • Consistent with other studies:

      • Andruski, Blumstein & Burton (1994)

      • Marslen-Wilson & Warren (1994)

      • Utman, Blumstein & Burton (2000)

      • Dahan, Magnuson, Tanenhaus & Hogan (2001)

      • McMurray, Clayards, Aslin & Tanenhaus (2004)

      • McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep)


    From sound to sense and back again the integration of lexical and speech processes

    CAD affects higher-level processes.

    From other work:

    Lexical activation influences sublexical representations.

    The Standard Paradigm?

    Sense

    Words

    Phonology

    Phonemes

    Samuel & Pitt (2003)

    Magnuson, McMurray, Tanehaus & Aslin (2003)

    Samuel (1997)

    Elman & McClelland (1988)

    Continuous Acoustic Detail


    From sound to sense and back again the integration of lexical and speech processes

    CAD affects higher-level processes.

    The Standard Paradigm?

    Sense

    From other work:

    Words

    Lexical activation influences sublexical representations.

    Phonology

    Phonemes

    Phonological regularity affects

    signal interpretation.

    Continuous Acoustic Detail

    Massaro & Cohen (1983)

    Halle, Segui, Frauenfelder & Meunier (1998)

    Pitt (1998)

    Dupoux,Kakehi, Hirose, Pallier & Mehler, (1999)


    From sound to sense and back again the integration of lexical and speech processes

    Sense

    Perhaps interaction and integration make sense.

    Do they help solve sticky problems?

    ?

    Words

    Phonology

    Phonemes

    YES

    Continuous Acoustic Detail


    From sound to sense and back again the integration of lexical and speech processes

    The Emerging Paradigm

    • Integration of work in:

      • spoken word recognition

      • speech perception

      • phonology

    • New computations simplify old problems and solve new ones.

      • Cognitive processes: Lexical activation & competition.

      • Perceptual processes: sensitivity to CAD & perceptual grouping.


    From sound to sense and back again the integration of lexical and speech processes

    • CAD is helpful in language comprehension.

      • Word segmentation

      • Coping with lawful variability due to assimilation

    • Combination of approaches helps solve both problems.


    From sound to sense and back again the integration of lexical and speech processes

    Lexical Segmentation

    Some lexical processes can’t work in the Standard Paradigm


    From sound to sense and back again the integration of lexical and speech processes

    The SWR Solution

    [  k t I v d I p A  t m I n t]


    From sound to sense and back again the integration of lexical and speech processes

    [  k t I v d I p A  t m I n t]

    active


    From sound to sense and back again the integration of lexical and speech processes

    [  k t I vd I p A  t m I n t]

    activedepartment


    From sound to sense and back again the integration of lexical and speech processes

    [  k t I v d I p A  t m I n t]

    activedepartment

    actof dip artmint

    apart

    departin

    are

    par

    Standard Paradigm: Template matching overgenerates


    From sound to sense and back again the integration of lexical and speech processes

    succeed

    suck

    activation

    seed

    ‘ k s I d -

    Cycle

    Frauenfelder & Peeters (1990)

    • Overgeneration resolved through competition in

    • TRACE (McClelland & Elman 1986)

    Problem: What if the speaker is trying to say “suck seeds”?


    From sound to sense and back again the integration of lexical and speech processes

    Words

    Implied processing model requires separate

    segmentation process

    Segmentation

    Phonemes

    Recognition

    CAD

    The Speech Solution

    • Cues shown to affect segmentation:

    • Initial strong syllable

    • Initial lengthening

    • Increased aspiration

    • Increased glottalization

    Lehiste, 1960; Garding,1967; Lehiste, 1972;

    Umeda, 1975; Nakatani & Dukes, 1977;

    Nakatani & Schaffer,1978; Cutler & Norris, 1988…..


    From sound to sense and back again the integration of lexical and speech processes

    Words

    Segmentation

    Phonemes

    Recognition

    CAD

    Problem: cues are subtle and varied,

    extra-segmental processes are inelegant

    ?

    Is there a better mechanism?


    From sound to sense and back again the integration of lexical and speech processes

    Syntax

    Syntax

    GRAMMAR primed

    GRAMMAR primed

    Tax

    INCOME inhibited

    Tax

    INCOME primed

    Gow & Gordon (1995)

    The proposal had a strange syntax that nobody liked.

    ^

    The proposal had a strange sin tax that nobody liked.

    ^

    • CAD affects interpretation.

    • does not trigger segmentation.


    From sound to sense and back again the integration of lexical and speech processes

    Good Start Model

    • Observation: All segmentation cues happen to enhance

    • word-initial features

      • Strengthened cues facilitate activation, making

      • intended words stronger competitors

    • Incorporating CAD:

    • Solves overgeneration problem.

    • No extra-segmental segmentation process.

    Gow & Gordon (1995)


    From sound to sense and back again the integration of lexical and speech processes

    Summary

    When continuous acoustic detail affects lexical activation, speech and SWR models can be integrated and simplified


    From sound to sense and back again the integration of lexical and speech processes

    Assimilation

    The emerging paradigm reframes computational problems


    From sound to sense and back again the integration of lexical and speech processes

    ripe berries?

    [  a I p ]# berries

    right berries?

    [ G  I m]# berries

    nonword?

    Redefining Computational Problems

    • English coronal place assimilation

    • /coronal # labial/ [labial # labial]

    • /coronal #velar/ [velar # velar]

    • Standard Paradigm: Change is

      • discrete

      • phonemically neutralizing


    From sound to sense and back again the integration of lexical and speech processes

    ripe

    Standard Paradigm solution: Phonological inference

    (Gaskell & Marslen-Wilson, 1996; 1998; 2001)

    Knowledge driven inference:

    If [labial # labial] infer /coronal # labial/

    • greem beans  green (Gaskell & Marslen-Wilson, 1996; Gow, 2001)

    ripe berries  right (Gaskell & Marslen-Wilson, 2001; Gow, 2002)

    Moreover: Assimilation effects dissociated from linguistic

    knowledge (Gow & Im, in press)


    From sound to sense and back again the integration of lexical and speech processes

    F3 Transitions in /æC/

    Contexts

    2800

    2750

    coronal

    2700

    assimilated

    Frequency (Hz)

    2650

    labial

    2600

    2550

    Pitch Period

    Assimilation Produces CAD

    Assimilatory modification is acoustically continuous

    F2 Transitions in /æC/

    Contexts

    1850

    1800

    1750

    coronal

    Frequency (Hz)

    1700

    assimilated

    labial

    1650

    1600

    1550

    Pitch Period

    This is not discrete feature change!


    From sound to sense and back again the integration of lexical and speech processes

    Regressive Context Effects

    Sma

    Select the

    catp box


    From sound to sense and back again the integration of lexical and speech processes

    Subject Hears: Assim_Non-Coronal (cat/p box)

    0.6

    0.5

    0.4

    Fixation Proportion

    0.3

    0.2

    Coronal (cat)

    0.1

    Non-Coronal (cap)

    0

    0

    400

    800

    1200

    1600

    Time (ms)


    From sound to sense and back again the integration of lexical and speech processes

    Subject Hears: Assim Non-Coronal (cat/p drawing)

    0.6

    0.5

    0.4

    Fixation Proportion

    0.3

    0.2

    Coronal (cat)

    Non-Coronal (cap)

    0.1

    0

    0

    400

    800

    1200

    1600

    Time (ms)


    From sound to sense and back again the integration of lexical and speech processes

    Progressive Context Effects

    Progressive effect in the same experiment


    From sound to sense and back again the integration of lexical and speech processes

    Assimilation: Use of CAD

    Assimilation is resolved through phonological context.

    Partially-assimilated items show

    regressive context effects (Gow, 2002; 2003)

    progressive context effects (Gow, 2001; 2003)

    Fully assimilated items show neither*

    (Gaskell & Marslen-Wilson, 2001; Gow, 2002;2003)


    From sound to sense and back again the integration of lexical and speech processes

    assimilation # context

    Infinite regress (eternal ambiguity)….

    or something more interesting?


    From sound to sense and back again the integration of lexical and speech processes

    Continuous acoustic detail is subject to basic perceptual processes


    From sound to sense and back again the integration of lexical and speech processes

    A Perceptual Account

    Feature cue parsing (Gow, 2003)

    [ k  t p b l E d ]


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)

    Features encoded by multiple cues that are integrated


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)

    Assimilation creates cues consistent with multiple places


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)

    Extract feature cues


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)

    Group feature cues by similarity and resolve ambiguity


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)

    example: eight….

    catp# box catp# drawing catp# 

    | | | |

    [cor] [cor] [COR] [cor]

    [lab] [LAB] [lab] [lab]


    From sound to sense and back again the integration of lexical and speech processes

    Feature cue parsing (Gow, 2003)

    example: eight….

    catp# Box catp# Drawing catp# 

    | |

    [cor] [cor] [COR] [cor]

    [lab] [LAB] [lab] [lab]

    Progressive and regressive effects fall out of grouping


    From sound to sense and back again the integration of lexical and speech processes

    Summary

    SWR problem (eternal ambiguity) replaced by simpler

    perceptual problem

    CAD important in solution: processing obstacle facilitates perception.

    Integration of continuous perceptual features facilitates higher-level processes.

    Facilitation via core-word recognition mechanisms—no extra-segmental routines required.


    From sound to sense and back again the integration of lexical and speech processes

    • The basis of the standard paradigm is undercut.

      • Meaning-based processes are affected by CAD.

      • CAD is an essential component of word recognition.

    The Standard Paradigm

    • Standard paradigm

      • Created artificial boundaries that misframed issues.

      • Continous acoustic detail is variability to be conquered..


    From sound to sense and back again the integration of lexical and speech processes

    The Emerging Paradigm

    • The emerging paradigm

      • Emphasis on methodologies that tap the minimal computational problem: meaning.

      • Stresses integration of speech and spoken word recognition, questions methods and theory.

      • Continuous acoustic detail is useful signal, not noise.


    From sound to sense and back again the integration of lexical and speech processes

    From Sound to Sense and back again:

    The integration of lexical and speech processes

    David Gow

    Massachusetts General Hospital

    Bob McMurray

    Dept. of Brain and Cognitive Sciences

    University of Rochester


  • Login