Geppeto 1 a modeling approach to study the production of speech gestures
Download
1 / 53

GEPPETO 1 : A modeling approach to study the production of speech gestures - PowerPoint PPT Presentation


  • 95 Views
  • Uploaded on

GEPPETO 1 : A modeling approach to study the production of speech gestures. Pascal Perrier (ICP – Grenoble) with Stéphanie Buchaillard (PhD) Matthieu Chabanas (ICP) Ma Liang (PhD), Yohan Payan (TIMC – Grenoble).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' GEPPETO 1 : A modeling approach to study the production of speech gestures' - maris


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Geppeto 1 a modeling approach to study the production of speech gestures

GEPPETO1: A modeling approach to study the production of speech gestures

Pascal Perrier (ICP – Grenoble)

with

Stéphanie Buchaillard (PhD)

Matthieu Chabanas (ICP)

Ma Liang (PhD),

Yohan Payan (TIMC – Grenoble)

1GEstures shaped by thePhysicsand by aPErceptuallyorientedTargetsOptimization


Outline
Outline

  • Introduction

  • Current hypotheses implemented in GEPPETO

  • Some results obtained with a 2D biomechanical tongue model

  • New issues raised by the use of 3D biomechanical tongue model


Basic issues in speech production research
Basic issuesin Speech Production Research

  • Phonology/Phonetics Interface

    • Link between discrete representations and continuous physical signals

    • Nature of physical correlates of speech units


Basic issues in speech production research1
Basic issuesin Speech Production Research

  • Control and Production of Speech Gestures

    • Control variables

    • Central representations of physical characteristics of the speech production apparatus

    • Interaction Perception-Action


Basic issues in speech production research2
Basic issuesin Speech Production Research

  • From Gestures to Speech Sounds

    • Nature of acoustic sources

    • Relations between motor commands and acoustics

    • Interaction between airflow and articulatory gestures.


What is geppeto
What is GEPPETO?

  • An evolutive modeling framework to quantitatively test hypotheses about the control and the production of speech gestures.

  • It includes

    • Hypotheses about the physical correlates of phonological units.

    • Models of motor control

    • Physical models of the speech production apparatus


Current hypotheses
Current Hypotheses

  • Phonology/Phonetic Interface

    • The smallest phonological unit is the phoneme

    • Phonemes are associated with target regions in the auditory domain

    • Larger phonological units are associated with speech sequences for which specific constraints exist for target optimization or for motor commands sequencing


Current hypotheses1
Current Hypotheses

  • Control of speech gestures

    • Control variables: l commands (EP Hypothesis, Feldman, 1966)

    • No on line use of feedback going through the cortex.

    • Short-delay orosensory and proprioceptive feedbacks are taken into account.

    • Existence in the brain of internal representations of the speech apparatus (internal models).


Current hypotheses2
Current Hypotheses

  • Control of speech gestures

    • Internal representations do not account for the whole physical complexity of the speech production apparatus

    • Kinematic characteristics are not directly controlled. They are the results of the interaction between motor control setups and physical phenomena of speech production

      • Which characteristics of speech signals are specifically controlled?


Application to the generation of speech gestures with a 2 d biomechanical tongue model
Application to the generation of speech gestures with a 2 D biomechanical tongue model

  • Implementation of the model of control

  • Inversion from desired perceptual objectives to motor commands

  • Generation of gestures


2d biomechanical model
2D Biomechanical Model biomechanical tongue model

  • Finite element structure

  • Linear elasticity (small deformations)

  • No account of the gravity


2d biomechanical model1

Posterior genioglossus biomechanical tongue model

Anterior Genioglossus

Hyoglossus

2D Biomechanical Model


2d biomechanical model2

Styloglossus biomechanical tongue model

Verticalis

Inferior Longitudinalis

2D Biomechanical Model


Learning a static internal model from l commands to formants
Learning a static internal model biomechanical tongue modelFrom l commands to formants

  • Step 1:

  • - Uniform sampling of

  • the l commands space

  • Generation of the

  • corresponding tongue

  • shapes.

9000 simulations


Learning a static internal model from l commands to formants1
Learning a static internal model biomechanical tongue modelFrom l commands to formants

Step 2: Computation of the area function.


Learning a static internal model from l commands to formants2
Learning a static internal model biomechanical tongue modelFrom l commands to formants

Step 3: Formants computation for 2 lip apertures

(red dots: spread lips; blue dots: rounded lips)


Learning a static internal model from l commands to formants3

1 biomechanical tongue modelst layer

Learning a static internal modelFrom l commands to formants

Step 4: Learning and generalizing

with radial basis functions

2nd layer


Inversion from target regions to l commands

Target regions for some non rounded French phonemes biomechanical tongue model

InversionFrom target regions to l commands

  • Target regions

  • Dispersion ellipses in the (F1, F2, F3) space

  • Currently defined by Fc1, Fc2, Fc3and sF1,sF2, sF3


Inversion from target regions to l commands1

Target regions for some non rounded French phonemes biomechanical tongue model

InversionFrom target regions to l commands

  • Target regions

  • Dispersion ellipses in the (F1, F2, F3) space

  • Currently defined by Fc1, Fc2, Fc3and sF1,sF2, sF3


Inversion from target regions to l commands2

+ biomechanical tongue model

Speaker oriented

Listener oriented

InversionFrom target regions to l commands

Optimization

Cost minimization (Gradient descent technique)

Cost for a sequence made of N phonemes

with


Inversion from target regions to l commands3
Inversion biomechanical tongue modelFrom target regions to l commands

Example 1 Sequence [œ-e-k-i]


Inversion from target regions to l commands4
Inversion biomechanical tongue modelFrom target regions to l commands

Example 2 Sequence [œ-e-k-a]


Production of tongue movements from inferred l commands

[oe] [e] [k] [a] biomechanical tongue model

Production of tongue movements from inferred l commands

Serial command patterns

No difference between vowels and consonants


Execution of tongue movements from inferred l commands
Execution of tongue movements from inferred biomechanical tongue modell commands

Öhman’s model: Vowel-to-Vowel basis

Consonants are seen as perturbation of V-V

[oe] [e] [k] [a]


Execution of tongue movements from inferred l commands1
Execution of tongue movements from inferred biomechanical tongue modell commands

Observed flesh point


Production of tongue movements from inferred l commands1

[a biomechanical tongue model]

[i]

Production of tongue movements from inferred l commands

Serial command patterns


Production of tongue movements from inferred l commands2

[a biomechanical tongue model]

[i]

Production of tongue movements from inferred l commands

Öhman’s command patterns


Interaction control physics influence on the shapes of the articulatory paths
Interaction control biomechanical tongue model/ physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

[aka]

[ika]

R. Houde (1969)


Fluid wall interaction
Fluid-Wall Interaction biomechanical tongue model

Imposed

pressure

difference

Forces

Mechanics of the

tissues.

Flow model

Finite

element model)

Deformation


Interaction control physics influence on the shapes of the articulatory paths1

Deplacement X - Y biomechanical tongue model

120

115

110

105

Y - mm

100

+++ PS = 3000 Pa

...... PS = 800 Pa

95

-------No aerodynamics

90

40

50

60

70

80

90

100

110

120

X - mm

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

[aka]


Interaction control physics influence on the shapes of the articulatory paths2
Interaction control biomechanical tongue model/ physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

No aerodynamics

With aerodynamics

[aka]


Interaction control physics influence on the shapes of the articulatory paths3

Deplacement X - Y biomechanical tongue model

113

112

... PS = 1600 Pa

---- No aerodynamics

111

110

Y - mm

109

108

107

61

62

63

64

65

66

67

X - mm

Interaction control / physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

[ika]


Interaction control physics influence on the shapes of the articulatory paths4
Interaction control biomechanical tongue model/ physics.Influence on the shapes of the articulatory paths

Example: the Articulatory loops

No aerodynamics

With aerodynamics

[ika]


A 3d biomechanical tongue model for a better account of physics
A 3D biomechanical tongue model: biomechanical tongue modelFor a better account of physics

  • Visible Human Project ® data

    (Wilhelms-Tricarico, 2003)

  • Finite Element Mesh made of Hexahedres

  • Adaptation of the mesh to a specific speaker (PB)

Gerard et al., ICP Grenoble

Wilhelms-Tricarico R.,1995


Inner muscle structure of the tongue
Inner muscle structure biomechanical tongue modelof the tongue

Genioglossus (medium)

Genioglossus (anterior)

Styloglossus

Geniohyoid

Genioglossus (posterior)

Hyoglossus

Verticalis

Transversus

Inferior longitudinalis

Mylohyoid

Superior longitudinalis


Vocal tract structure
Vocal tract structure biomechanical tongue model

TONGUE’S BODY

HYOID BONE

MANDIBLE

PALATE

OTHER MUSCLES


Elastical properties of tongue muscles

Displacement biomechanical tongue model

0

Force

Linear

Non Linear

Tongue

Indentator

Elastical properties of tongue muscles

  • Hyperelastic material (2nd order Yeoh model) with large deformation hypothesis


Effect of gravity
Effect of gravity biomechanical tongue model

[1s]


Dealing with gravity with the ep hypothesis
Dealing with gravity with the EP hypothesis biomechanical tongue model

[300ms]


Dealing with gravity with the ep hypothesis1
Dealing with gravity with the EP hypothesis biomechanical tongue model

  • Activation of GGp and MH

  •  Increase of reflex activity

[300ms]


Dealing with gravity with the ep hypothesis2
Dealing with gravity with the EP hypothesis biomechanical tongue model

GGP activation


Dealing with gravity with the ep hypothesis3
Dealing with gravity with the EP hypothesis biomechanical tongue model

Example of a good choice of control parameters

[300ms]


Conclusions
Conclusions biomechanical tongue model

  • A model of control based on perceptual objectives specified in terms of formants target regions associated with l motor commands and on an optimization process using a static model of the motor-perception relations can generate realistic speech movements if it is applying to a realistic physical model of speech production.


Conclusions1
Conclusions biomechanical tongue model

  • It supports our hypothesis that there is not need to assume the existence of a central optimization process that would apply to the articulatory trajectories in their whole (i.e. minimum of jerk, minimum of torque…)


Conclusions2
Conclusions biomechanical tongue model

  • It gives an interesting account of coarticulation phenomena by separating the effects of planning and those of physics.

  • It permits to test hypotheses about the phonological units (see serial model versus Öhman’s model).


Conclusions3
Conclusions biomechanical tongue model

However

  • a systematic comparison with data is required (currently in progress for French, German, Chinese, Japanese)

  • No account for time control, or for hypo/hyperspeech

  • No account for gravity


Conclusions4
Conclusions biomechanical tongue model

  • Necessity to work on a more complex internal representations that would integrate some aspects of articulatory dynamics.


Thank you biomechanical tongue model


Influence of elasticity modeling
Influence of elasticity modeling biomechanical tongue model

Hyperelastic

Large defo.

Linear

Small defo.

Linear

Activation of the Hyoglossus (2N)


Ep hypothesis feldman 1966
EP Hypothesis biomechanical tongue model(Feldman, 1966)

Perrier, Ostry, Laboissière, 1996


Ep hypothesis feldman 19661
EP Hypothesis biomechanical tongue model (Feldman, 1966)

Perrier, Ostry, Laboissière, 1996


Static internal models

l biomechanical tongue modeld

Inverse Model

yi(t)

Direct Model

Static Internal Models

Central Nervous System

Desired

formants

Peripheral motor

system

l

Formants


ad