Geppeto 1 a modeling approach to study the production of speech gestures
This presentation is the property of its rightful owner.
Sponsored Links
1 / 53

GEPPETO 1 : A modeling approach to study the production of speech gestures PowerPoint PPT Presentation


  • 67 Views
  • Uploaded on
  • Presentation posted in: General

GEPPETO 1 : A modeling approach to study the production of speech gestures. Pascal Perrier (ICP – Grenoble) with Stéphanie Buchaillard (PhD) Matthieu Chabanas (ICP) Ma Liang (PhD), Yohan Payan (TIMC – Grenoble).

Download Presentation

GEPPETO 1 : A modeling approach to study the production of speech gestures

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Geppeto 1 a modeling approach to study the production of speech gestures

GEPPETO1: A modeling approach to study the production of speech gestures

Pascal Perrier (ICP – Grenoble)

with

Stéphanie Buchaillard (PhD)

Matthieu Chabanas (ICP)

Ma Liang (PhD),

Yohan Payan (TIMC – Grenoble)

1GEstures shaped by thePhysicsand by aPErceptuallyorientedTargetsOptimization


Outline

Outline

  • Introduction

  • Current hypotheses implemented in GEPPETO

  • Some results obtained with a 2D biomechanical tongue model

  • New issues raised by the use of 3D biomechanical tongue model


Basic issues in speech production research

Basic issuesin Speech Production Research

  • Phonology/Phonetics Interface

    • Link between discrete representations and continuous physical signals

    • Nature of physical correlates of speech units


Basic issues in speech production research1

Basic issuesin Speech Production Research

  • Control and Production of Speech Gestures

    • Control variables

    • Central representations of physical characteristics of the speech production apparatus

    • Interaction Perception-Action


Basic issues in speech production research2

Basic issuesin Speech Production Research

  • From Gestures to Speech Sounds

    • Nature of acoustic sources

    • Relations between motor commands and acoustics

    • Interaction between airflow and articulatory gestures.


What is geppeto

What is GEPPETO?

  • An evolutive modeling framework to quantitatively test hypotheses about the control and the production of speech gestures.

  • It includes

    • Hypotheses about the physical correlates of phonological units.

    • Models of motor control

    • Physical models of the speech production apparatus


Current hypotheses

Current Hypotheses

  • Phonology/Phonetic Interface

    • The smallest phonological unit is the phoneme

    • Phonemes are associated with target regions in the auditory domain

    • Larger phonological units are associated with speech sequences for which specific constraints exist for target optimization or for motor commands sequencing


Current hypotheses1

Current Hypotheses

  • Control of speech gestures

    • Control variables: l commands (EP Hypothesis, Feldman, 1966)

    • No on line use of feedback going through the cortex.

    • Short-delay orosensory and proprioceptive feedbacks are taken into account.

    • Existence in the brain of internal representations of the speech apparatus (internal models).


Current hypotheses2

Current Hypotheses

  • Control of speech gestures

    • Internal representations do not account for the whole physical complexity of the speech production apparatus

    • Kinematic characteristics are not directly controlled. They are the results of the interaction between motor control setups and physical phenomena of speech production

      • Which characteristics of speech signals are specifically controlled?


Application to the generation of speech gestures with a 2 d biomechanical tongue model

Application to the generation of speech gestures with a 2 D biomechanical tongue model

  • Implementation of the model of control

  • Inversion from desired perceptual objectives to motor commands

  • Generation of gestures


2d biomechanical model

2D Biomechanical Model

  • Finite element structure

  • Linear elasticity (small deformations)

  • No account of the gravity


2d biomechanical model1

Posterior genioglossus

Anterior Genioglossus

Hyoglossus

2D Biomechanical Model


2d biomechanical model2

Styloglossus

Verticalis

Inferior Longitudinalis

2D Biomechanical Model


Learning a static internal model from l commands to formants

Learning a static internal modelFrom l commands to formants

  • Step 1:

  • - Uniform sampling of

  • the l commands space

  • Generation of the

  • corresponding tongue

  • shapes.

9000 simulations


Learning a static internal model from l commands to formants1

Learning a static internal modelFrom l commands to formants

Step 2: Computation of the area function.


Learning a static internal model from l commands to formants2

Learning a static internal modelFrom l commands to formants

Step 3: Formants computation for 2 lip apertures

(red dots: spread lips; blue dots: rounded lips)


Learning a static internal model from l commands to formants3

1st layer

Learning a static internal modelFrom l commands to formants

Step 4: Learning and generalizing

with radial basis functions

2nd layer


Inversion from target regions to l commands

Target regions for some non rounded French phonemes

InversionFrom target regions to l commands

  • Target regions

  • Dispersion ellipses in the (F1, F2, F3) space

  • Currently defined by Fc1, Fc2, Fc3and sF1,sF2, sF3


Inversion from target regions to l commands1

Target regions for some non rounded French phonemes

InversionFrom target regions to l commands

  • Target regions

  • Dispersion ellipses in the (F1, F2, F3) space

  • Currently defined by Fc1, Fc2, Fc3and sF1,sF2, sF3


Inversion from target regions to l commands2

+

Speaker oriented

Listener oriented

InversionFrom target regions to l commands

Optimization

Cost minimization (Gradient descent technique)

Cost for a sequence made of N phonemes

with


Inversion from target regions to l commands3

InversionFrom target regions to l commands

Example 1 Sequence [œ-e-k-i]


Inversion from target regions to l commands4

InversionFrom target regions to l commands

Example 2 Sequence [œ-e-k-a]


Production of tongue movements from inferred l commands

[oe] [e] [k] [a]

Production of tongue movements from inferred l commands

Serial command patterns

No difference between vowels and consonants


Execution of tongue movements from inferred l commands

Execution of tongue movements from inferred l commands

Öhman’s model: Vowel-to-Vowel basis

Consonants are seen as perturbation of V-V

[oe] [e] [k] [a]


Execution of tongue movements from inferred l commands1

Execution of tongue movements from inferred l commands

Observed flesh point


Production of tongue movements from inferred l commands1

[a]

[i]

Production of tongue movements from inferred l commands

Serial command patterns


Production of tongue movements from inferred l commands2

[a]

[i]

Production of tongue movements from inferred l commands

Öhman’s command patterns


Interaction control physics influence on the shapes of the articulatory paths

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

[aka]

[ika]

R. Houde (1969)


Fluid wall interaction

Fluid-Wall Interaction

Imposed

pressure

difference

Forces

Mechanics of the

tissues.

Flow model

Finite

element model)

Deformation


Interaction control physics influence on the shapes of the articulatory paths1

Deplacement X - Y

120

115

110

105

Y - mm

100

+++ PS = 3000 Pa

...... PS = 800 Pa

95

-------No aerodynamics

90

40

50

60

70

80

90

100

110

120

X - mm

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

[aka]


Interaction control physics influence on the shapes of the articulatory paths2

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

No aerodynamics

With aerodynamics

[aka]


Interaction control physics influence on the shapes of the articulatory paths3

Deplacement X - Y

113

112

... PS = 1600 Pa

---- No aerodynamics

111

110

Y - mm

109

108

107

61

62

63

64

65

66

67

X - mm

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

[ika]


Interaction control physics influence on the shapes of the articulatory paths4

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

No aerodynamics

With aerodynamics

[ika]


A 3d biomechanical tongue model for a better account of physics

A 3D biomechanical tongue model:For a better account of physics

  • Visible Human Project ® data

    (Wilhelms-Tricarico, 2003)

  • Finite Element Mesh made of Hexahedres

  • Adaptation of the mesh to a specific speaker (PB)

Gerard et al., ICP Grenoble

Wilhelms-Tricarico R.,1995


Inner muscle structure of the tongue

Inner muscle structure of the tongue

Genioglossus (medium)

Genioglossus (anterior)

Styloglossus

Geniohyoid

Genioglossus (posterior)

Hyoglossus

Verticalis

Transversus

Inferior longitudinalis

Mylohyoid

Superior longitudinalis


Vocal tract structure

Vocal tract structure

TONGUE’S BODY

HYOID BONE

MANDIBLE

PALATE

OTHER MUSCLES


Elastical properties of tongue muscles

Displacement

0

Force

Linear

Non Linear

Tongue

Indentator

Elastical properties of tongue muscles

  • Hyperelastic material (2nd order Yeoh model) with large deformation hypothesis


Effect of gravity

Effect of gravity

[1s]


Dealing with gravity with the ep hypothesis

Dealing with gravity with the EP hypothesis

[300ms]


Dealing with gravity with the ep hypothesis1

Dealing with gravity with the EP hypothesis

  • Activation of GGp and MH

  •  Increase of reflex activity

[300ms]


Dealing with gravity with the ep hypothesis2

Dealing with gravity with the EP hypothesis

GGP activation


Dealing with gravity with the ep hypothesis3

Dealing with gravity with the EP hypothesis

Example of a good choice of control parameters

[300ms]


Conclusions

Conclusions

  • A model of control based on perceptual objectives specified in terms of formants target regions associated with l motor commands and on an optimization process using a static model of the motor-perception relations can generate realistic speech movements if it is applying to a realistic physical model of speech production.


Conclusions1

Conclusions

  • It supports our hypothesis that there is not need to assume the existence of a central optimization process that would apply to the articulatory trajectories in their whole (i.e. minimum of jerk, minimum of torque…)


Conclusions2

Conclusions

  • It gives an interesting account of coarticulation phenomena by separating the effects of planning and those of physics.

  • It permits to test hypotheses about the phonological units (see serial model versus Öhman’s model).


Conclusions3

Conclusions

However

  • a systematic comparison with data is required (currently in progress for French, German, Chinese, Japanese)

  • No account for time control, or for hypo/hyperspeech

  • No account for gravity


Conclusions4

Conclusions

  • Necessity to work on a more complex internal representations that would integrate some aspects of articulatory dynamics.


Geppeto 1 a modeling approach to study the production of speech gestures

Thank you


Influence of elasticity modeling

Influence of elasticity modeling

Hyperelastic

Large defo.

Linear

Small defo.

Linear

Activation of the Hyoglossus (2N)


Ep hypothesis feldman 1966

EP Hypothesis(Feldman, 1966)

Perrier, Ostry, Laboissière, 1996


Ep hypothesis feldman 19661

EP Hypothesis (Feldman, 1966)

Perrier, Ostry, Laboissière, 1996


Static internal models

ld

Inverse Model

yi(t)

Direct Model

Static Internal Models

Central Nervous System

Desired

formants

Peripheral motor

system

l

Formants


  • Login