From Sound to Sense and back again:
Download
1 / 63

From Sound to Sense and back again: The integration of lexical and speech processes - PowerPoint PPT Presentation


  • 258 Views
  • Uploaded on

From Sound to Sense and back again: The integration of lexical and speech processes. David Gow Massachusetts General Hospital. Bob McMurray Dept. of Brain and Cognitive Sciences University of Rochester. Sense. Sound. The Speech Chain.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'From Sound to Sense and back again: The integration of lexical and speech processes' - albert


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

From Sound to Sense and back again:

The integration of lexical and speech processes

David Gow

Massachusetts General Hospital

Bob McMurray

Dept. of Brain and Cognitive Sciences

University of Rochester


Sense

Sound

The Speech Chain

Complex computations from sound to sense must be broken up for study.

Assume intermediate representations:

Phonemes…

Words…

Syntactic Phrases…


The Standard Paradigm

Sense

The Standard Paradigm

Words

Phonology

Phonemes

Sound


Phonemes*essential

* or other sublexical category

The Standard Paradigm

Sense

The Standard Paradigm

Delimited fields of study.

  • Speech Perception

Words

  • Spoken Word Recognition

Phonology

Phonemes

  • Phonology

Sound


100

100

Discrimination

% /p/

  • Sharp identification of tokens on a continuum.

Discrimination

ID (%/pa/)

0

0

B

VOT

P

  • Discrimination poor within a phonetic category.

Why?

Categorical Perception (CP)

Continuous Acoustic Detail => Discrete Categories

Does CAD affect speech categorization?


Sense

Categorical Perception (CP)

  • Defined fundamental computational problems.

  • CP is output of

    • Speech perception

  • Input to

    • Phonology

    • Word recognition.

Words

Phonology

Phonemes

Sound


CP

  • But…

    • Not all speech contrasts are categorical.

    • Lots of tasks show non-categorical perception.

Fry, Abramson, Eimas & Liberman (1962) Pisoni & Tash (1974) Pisoni & Lazarus (1974) Carney, Widden & Viemeister (1977) Hary & Massaro (1982) Pisoni, Aslin, Perey & Hennessy (1982) Healy & Repp (1982) Massaro & Cohen (1983) Miller (1997) Samuel (1997)…


Sense

?

CP

Words

CP tasks don’t necessarily tap a stage of this problem.

Sound

Lexical activation… seems a good bet.

Why has the Standard Paradigm persisted?

Categorical Perception is

about phonetic classification.

The minimal computational problem: compute meaning from sound.


Why has the Standard Paradigm persisted?

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.


Segmentation

Cue extra-segmental process.

Why has the Standard Paradigm persisted?

Even when continuous acoustic detail affects word recognition, it is seen as outside of core word recognition.

  • Example: Word Segmentation

    • Vowel Length

    • Stress/Meter

    • Coarticulation

Words

Phonemes

Word Recognition

CAD


Does continuous acoustic detail affect interpretation via core word-recognition processes?

  • No. Standard Paradigm is fine…

    • Yes. Hmm…

Sublexical Filter

(phonemes)

  • Need to use stimuli with:

    • Precise control over CAD

  • Need to use tasks that:

    • reflect only minimal computational problem: meaning.

    • are sensitive to acoustic detail.


Visual World Paradigm

Visual World Paradigm

  • Subjects hear spoken language and manipulate objects in a visual world.

  • Visual world includes set of objects with interesting linguistic properties (names)

  • Eye-movements to each object are monitored throughout the task.

Tanenhaus, Spivey-Knowlton, Eberhart & Sedivy (1995)

Allopenna, Magnuson & Tanenhaus (1998)


  • Meaning based, natural task: Subjects must interpret speech to perform task.

  • Fixation probability maps onto dynamics of lexical activation.

  • Context is controlled:

  • meaning  lexical activation.

    • Eye-movements fast and time-locked to speech.


    ?

    Does continuous acoustic detail affect interpretation?

    Is lexical activation sensitive to continuous acoustic detail?


    McMurray, Tanenhaus & Aslin (2003)

    • Combine tools of

      • speech perception:

        • 9-step VOT continuum.

    • spoken word recognition:

      • visual world paradigm


    Methods

    A moment to view the items



    Bear

    Repeat 1080 times…


    200 ms

    Trials

    1

    2

    3

    4

    5

    Time

    Target =Bear

    Competitor =Pear

    Unrelated =Lamp, Ship


    0.9

    0.8

    0.7

    0.6

    0.5

    0.4

    Fixation proportion

    0.3

    0.2

    0.1

    0

    0

    400

    800

    1200

    1600

    VOT=0 Response=

    Time (ms)


    target

    Fixation proportion

    Fixation proportion

    time

    time

    Predictions

    What would lexical sensitivity to CAD look like?

    Systematic effect on competitor dynamics.

    Fixations to the competitor.

    Categorical Results

    Gradient Effect

    target

    target

    competitor

    competitor

    competitor

    competitor


    20 ms

    25 ms

    30 ms

    10 ms

    15 ms

    35 ms

    40 ms

    0.16

    0.14

    0.12

    0.1

    0.08

    0.06

    0.04

    0.02

    0

    0

    400

    800

    1200

    1600

    0

    400

    800

    1200

    1600

    2000

    Results

    Response=

    Response=

    VOT

    VOT

    0 ms

    5 ms

    Competitor Fixations

    Time since word onset (ms)


    P

    B

    Sh

    L

    Task?

    Phoneme ID

    Not part of minimal

    computational problem.

    Same stimuli in

    metalinguistic task…

    …more categorical pattern of fixations

    Continuous acoustic detail is not helpful in metalinguistic tasks…


    Summary

    Word recognition shows gradient sensitivity to continuous acoustic detail.

    Not extra-segmental: VOT

    CAD affects higher-level processes.

    • Consistent with other studies:

      • Andruski, Blumstein & Burton (1994)

      • Marslen-Wilson & Warren (1994)

      • Utman, Blumstein & Burton (2000)

      • Dahan, Magnuson, Tanenhaus & Hogan (2001)

      • McMurray, Clayards, Aslin & Tanenhaus (2004)

      • McMurray, Aslin, Tanenhaus, Spivey & Subik (in prep)


    CAD affects higher-level processes.

    From other work:

    Lexical activation influences sublexical representations.

    The Standard Paradigm?

    Sense

    Words

    Phonology

    Phonemes

    Samuel & Pitt (2003)

    Magnuson, McMurray, Tanehaus & Aslin (2003)

    Samuel (1997)

    Elman & McClelland (1988)

    Continuous Acoustic Detail


    CAD affects higher-level processes.

    The Standard Paradigm?

    Sense

    From other work:

    Words

    Lexical activation influences sublexical representations.

    Phonology

    Phonemes

    Phonological regularity affects

    signal interpretation.

    Continuous Acoustic Detail

    Massaro & Cohen (1983)

    Halle, Segui, Frauenfelder & Meunier (1998)

    Pitt (1998)

    Dupoux,Kakehi, Hirose, Pallier & Mehler, (1999)


    Sense

    Perhaps interaction and integration make sense.

    Do they help solve sticky problems?

    ?

    Words

    Phonology

    Phonemes

    YES

    Continuous Acoustic Detail


    The Emerging Paradigm

    • Integration of work in:

      • spoken word recognition

      • speech perception

      • phonology

    • New computations simplify old problems and solve new ones.

      • Cognitive processes: Lexical activation & competition.

      • Perceptual processes: sensitivity to CAD & perceptual grouping.



    Lexical Segmentation

    Some lexical processes can’t work in the Standard Paradigm


    The SWR Solution

    [  k t I v d I p A  t m I n t]


    [  k t I v d I p A  t m I n t]

    active


    [  k t I vd I p A  t m I n t]

    activedepartment


    [  k t I v d I p A  t m I n t]

    activedepartment

    actof dip artmint

    apart

    departin

    are

    par

    Standard Paradigm: Template matching overgenerates


    succeed

    suck

    activation

    seed

    ‘ k s I d -

    Cycle

    Frauenfelder & Peeters (1990)

    • Overgeneration resolved through competition in

    • TRACE (McClelland & Elman 1986)

    Problem: What if the speaker is trying to say “suck seeds”?


    Words

    Implied processing model requires separate

    segmentation process

    Segmentation

    Phonemes

    Recognition

    CAD

    The Speech Solution

    • Cues shown to affect segmentation:

    • Initial strong syllable

    • Initial lengthening

    • Increased aspiration

    • Increased glottalization

    Lehiste, 1960; Garding,1967; Lehiste, 1972;

    Umeda, 1975; Nakatani & Dukes, 1977;

    Nakatani & Schaffer,1978; Cutler & Norris, 1988…..


    Words

    Segmentation

    Phonemes

    Recognition

    CAD

    Problem: cues are subtle and varied,

    extra-segmental processes are inelegant

    ?

    Is there a better mechanism?


    Syntax

    Syntax

    GRAMMAR primed

    GRAMMAR primed

    Tax

    INCOME inhibited

    Tax

    INCOME primed

    Gow & Gordon (1995)

    The proposal had a strange syntax that nobody liked.

    ^

    The proposal had a strange sin tax that nobody liked.

    ^

    • CAD affects interpretation.

    • does not trigger segmentation.


    Good Start Model

    • Observation: All segmentation cues happen to enhance

    • word-initial features

      • Strengthened cues facilitate activation, making

      • intended words stronger competitors

    • Incorporating CAD:

    • Solves overgeneration problem.

    • No extra-segmental segmentation process.

    Gow & Gordon (1995)


    Summary

    When continuous acoustic detail affects lexical activation, speech and SWR models can be integrated and simplified


    Assimilation

    The emerging paradigm reframes computational problems


    ripe berries?

    [  a I p ]# berries

    right berries?

    [ G  I m]# berries

    nonword?

    Redefining Computational Problems

    • English coronal place assimilation

    • /coronal # labial/ [labial # labial]

    • /coronal #velar/ [velar # velar]

    • Standard Paradigm: Change is

      • discrete

      • phonemically neutralizing


    ripe

    Standard Paradigm solution: Phonological inference

    (Gaskell & Marslen-Wilson, 1996; 1998; 2001)

    Knowledge driven inference:

    If [labial # labial] infer /coronal # labial/

    • greem beans  green (Gaskell & Marslen-Wilson, 1996; Gow, 2001)

    ripe berries  right (Gaskell & Marslen-Wilson, 2001; Gow, 2002)

    Moreover: Assimilation effects dissociated from linguistic

    knowledge (Gow & Im, in press)


    F3 Transitions in /æC/

    Contexts

    2800

    2750

    coronal

    2700

    assimilated

    Frequency (Hz)

    2650

    labial

    2600

    2550

    Pitch Period

    Assimilation Produces CAD

    Assimilatory modification is acoustically continuous

    F2 Transitions in /æC/

    Contexts

    1850

    1800

    1750

    coronal

    Frequency (Hz)

    1700

    assimilated

    labial

    1650

    1600

    1550

    Pitch Period

    This is not discrete feature change!


    Regressive Context Effects

    Sma

    Select the

    catp box


    Subject Hears: Assim_Non-Coronal (cat/p box)

    0.6

    0.5

    0.4

    Fixation Proportion

    0.3

    0.2

    Coronal (cat)

    0.1

    Non-Coronal (cap)

    0

    0

    400

    800

    1200

    1600

    Time (ms)


    Subject Hears: Assim Non-Coronal (cat/p drawing)

    0.6

    0.5

    0.4

    Fixation Proportion

    0.3

    0.2

    Coronal (cat)

    Non-Coronal (cap)

    0.1

    0

    0

    400

    800

    1200

    1600

    Time (ms)


    Progressive Context Effects

    Progressive effect in the same experiment


    Assimilation: Use of CAD

    Assimilation is resolved through phonological context.

    Partially-assimilated items show

    regressive context effects (Gow, 2002; 2003)

    progressive context effects (Gow, 2001; 2003)

    Fully assimilated items show neither*

    (Gaskell & Marslen-Wilson, 2001; Gow, 2002;2003)


    assimilation # context

    Infinite regress (eternal ambiguity)….

    or something more interesting?



    A Perceptual Account processes

    Feature cue parsing (Gow, 2003)

    [ k  t p b l E d ]


    Feature cue parsing processes (Gow, 2003)

    Features encoded by multiple cues that are integrated


    Feature cue parsing processes (Gow, 2003)


    Feature cue parsing processes (Gow, 2003)

    Assimilation creates cues consistent with multiple places


    Feature cue parsing processes (Gow, 2003)

    Extract feature cues


    Feature cue parsing processes (Gow, 2003)

    Group feature cues by similarity and resolve ambiguity


    Feature cue parsing processes (Gow, 2003)

    example: eight….

    catp# box catp# drawing catp# 

    | | | |

    [cor] [cor] [COR] [cor]

    [lab] [LAB] [lab] [lab]


    Feature cue parsing processes (Gow, 2003)

    example: eight….

    catp# Box catp# Drawing catp# 

    | |

    [cor] [cor] [COR] [cor]

    [lab] [LAB] [lab] [lab]

    Progressive and regressive effects fall out of grouping


    Summary processes

    SWR problem (eternal ambiguity) replaced by simpler

    perceptual problem

    CAD important in solution: processing obstacle facilitates perception.

    Integration of continuous perceptual features facilitates higher-level processes.

    Facilitation via core-word recognition mechanisms—no extra-segmental routines required.


    • The processes basis of the standard paradigm is undercut.

      • Meaning-based processes are affected by CAD.

      • CAD is an essential component of word recognition.

    The Standard Paradigm

    • Standard paradigm

      • Created artificial boundaries that misframed issues.

      • Continous acoustic detail is variability to be conquered..


    The Emerging Paradigm processes

    • The emerging paradigm

      • Emphasis on methodologies that tap the minimal computational problem: meaning.

      • Stresses integration of speech and spoken word recognition, questions methods and theory.

      • Continuous acoustic detail is useful signal, not noise.


    From Sound to Sense and back again: processes

    The integration of lexical and speech processes

    David Gow

    Massachusetts General Hospital

    Bob McMurray

    Dept. of Brain and Cognitive Sciences

    University of Rochester


    ad