Pascal Perrier 1 , Jean-François Patri 1,2 , Julien Diard 2

1. From abstract phonemes to speech movements: the role of orofacial biomechanics and multisensory motor goals. Pascal Perrier1, Jean-François Patri1,2, Julien Diard2 1.GIPSA-lab, CNRS, Univ. Grenoble Alpes, France 2.LPNC, CNRS, Univ. Grenoble Alpes, France

2. Jean-François Patri, PhD Student – Gipsa-lab Julien Diard CNRS Researcher – LPNC

3. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Other collaborators • Yohan Payan – TIMC Grenoble • Susanne Fuchs – ZAS – Berlin • Jean-Luc Schwartz (Gipsa-lab) • Liang Ma (Shangai University) • Mohammad Ali Nazari – Tehran University • Postdocs: Ralf Winkler (2010 – 2012) • Former PhD Students: Stéphanie Buchaillard (2007), Jean-Michel Gérard (2004)

4. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Introduction • An old issue • Numerous contributions • Articulatory system • Perception system • Memory • Dynamics • Still unsolved • Phonology/Phonetics • Invariance/Variability • Abstraction/Physics • Categorical/Continuous

5. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Some unsolved issues • Nature of the units (Phonemes, Syllables…) • Nature of the physical correlates of phonological categories • Auditory/Somatosensory/Visual • Discrete/Continuous • Acceptable variability • Variation in variability (Phonetic Convergence, Aging…) My conviction: in thesestudies, while the account of the speech perception and memorysystems go beyondwhatisdirectlymeasurable, the account of speech production remainsat the level of measureable articulation and acoustics.

6. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Integrating more complex speech production mechanims • Orofacial biomechanics • Soft tissues • Muscle mechanics • Motor control • Variables of control • Sensory feedback • Planning (Anticipation) • Feedback control / Feedforward control

7. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Outline • Introduction • A model of speech production: GEPPETO • Tongue biomechanics and the vowelspace • Vowelreduction and tonguebiomechanics • Plasticity in speech production and sensorypreference • Conclusion,

8. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion A Model of speech production: GEPPETO1 Planning model m o d E l Motor control model Physical models 1holds for "GEstures shaped by the Physics and by a PErceptually oriented Targets Optimization

9. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Biomechanical models 3D model of the jaw, the face and the tongue Gerard et al., 2006; Buchaillard et al., 2009, Nazari et al., 2011, Stavness et al., 2013

10. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Biomechanical models 2D model of the tongue Payan & Perrier, 1996; Perrier et al., 2003

11. Sagittal view  Area Function Perrier et al., 1991 Area Function  Formant & Signal Story, 2005 Badin & Fant, 1984 Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Articulatory to acoustic model 2D model only F3 F1 F2

12. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Motor control model: the l model Control variables Threshold muscle length l Shift between Equilibrium Positions at a constant shift of rate Feldman, 1986; Perrier et al., 1996; Perrier et al., 2003

13. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Planning model: a Bayesian approach

14. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Planning model: a Bayesian approach Internal models Motor goals

15. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Front Raising and Back Raising Factors Harshman et al., JASA 1977 • Icelandic (Nix et al., JASA 1996, correcting Jackson, JASA 1998) • German (Hoole, JASA 1999) • French (Maeda, JEPs1979; Beautemps et al., JASA 2001) • Japanese (Mokhtari et al., J. Phon 2007

16. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Posterior genioglossus Anterior Genioglossus Hyoglossus Styloglossus Verticalis Inferior Longitudinalis 2D Tongue Model (Payan & Perrier, 1996; Perrier et al, 2003)

17. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Uniform Sampling of the Motor space Perrier et al, ICSLP 2000, Perrier et al, Interspeech 2005 • 6 muscles (GGP, GGA, STY, HYO, IL, VERT) • 8800 simulations Sagittal view

18. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Principal Component AnalysisPerrier et al, ICSLP 2000; Fuchs & Perrier, ZASPIL 2005 F1 = 69 % of the variance F1+F2 = 88 % of the variance

19. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Principal Component AnalysisPerrier et al, ICSLP 2000; Fuchs & Perrier, ZASPIL 2005 F1+F2+F3= 96 % of the variance F1+F2+F3+F4= 99 %

20. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Tongue biomechanics and vowel space • Muscle anatony determines the degrees of freedom in vowel production • Influences the main directions of variability • Influences the way similar articulatory shapes and acoustic properties can be obtained with variables motor commands (Motor equivalence strategies) • Muscle anatomy contributes to determine the structure of the vowel systems

21. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Vowel reduction Lindblom, 1963; Lindblom, 1967

22. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Variability in Speech Production Three spaces Patri et al., 2016

23. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Variability in Speech Production Two types of constraint Auditory region Force level (Cocontraction) Patri et al., 2016

24. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Bayesian Planning model Patri et al., 2015 ; 2016

25. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Bayesian Planning model : combining 2 constraints Patri et al., 2016

26. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Bayesian Planning model : combining 2 constraints Patri et al., 2016 F1 [Hz] (reversed) F2 [Hz] (reversed)

27. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Bayesian Planning Model: Sequence Patri et al., 2015; 2016 Motor constraint  Anticipatory coarticulation

28. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Planning of Sequence /aiE/ Patri et al., 2016 Normal Fast Normal Fast 4 Conditions 22

29. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Sequence /aiE/ - Weak force level Patri et al., 2016 Fast Normal Weak Reduction Clarity loss [Thanks to Brad Story for the acoustical synthesis]

30. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Sequence /aiE/ - Strong force level Patri et al., 2016 Fast Normal Strong No Reduction [Thanks to Brad Story for the acoustical synthesis]

31. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Conclusion • Vowel reduction can be controlled (Hypo versus Hyperspeech, Lindblom 1990) • No need to change the underlying articulatory and acoustic goals: can results from the timing and articulatory dynamics • Muscle biomechanics explains how to control articulatory dynamics • Confronting simulations and data can help understanding the phonological reduction of vowel to schwa

32. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Plasticity of speech production • Speech production changes due to the interaction with other speakers • Phonetic convergence (Pardo, 2006) • L2 learning • However the magnitude of the changes is highly speaker dependent (Pardo, 2013) • An interesting experimental paradigm: Perturbation of the auditory feedback (Houde & Jordan, 1998) (Shiller, in this conference) • Compensation is never complete • Large variability across speakers in compensation magnitude

33. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Sensory preferenceLametti et al, 2012 Somatosensory perturbation + Auditory perturbation

34. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Modeling sensory preference Patri et al., 2017

35. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Modeling sensory preference Patri et al., 2017 Acuity (or preference) increases

36. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Adaptation to altered auditory feedback Patri et al., 2017 Unperturbed condition Perturbed condition dF1 Result of adaptation : Change in the auditory-motor internal model

37. Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion Variability in the magnitude of the compensation due to sensory preference Patri et al., 2017 Somatosensory Auditory i e  E A

38. Conclusion Making assumptions about the nature of the motor goals (Auditory, Somatosensory) and accounting for sensory preference helps understanding subject-specific speech production plasticity May contribute to a better understanding of ontogenetic evolution of speech (speech development and aging) May contribute to a better understanding of language evolution Introduction Geppeto Vowelspace Vowelreduction Plasticity Conclusion

39. Thank You