Teaching Tool For French Speech Pronunciation

1. Teaching Tool For French Speech Pronunciation Capstone Design Project 2008 Joseph Ciaburri Advisor: Professor Catravas

2. Motivation Use feedback that allows for self diagnosis Make tool as simple as possible for student Improve French pronunciation through the repetition of visual and aural aids Tony Blair Congratulating Nicolas Sarkozy on Election Win Interview with Domnique Villepin

3. Data Acquisition Data Audio/Visual Databank Data Microphone Webcam Window 2 Audio and Video of User Speaking Window 1 Native Speaker Data Speech Video Audio Audio USER Window 3 Diagnostics Video Video Audio Video Proposed Learning System

4. Design Specifications Goals Read in audio and video at the same time Play back audio and video at the same time within 1 second Minimize system requirements Implement diagnostics that are sensitive to pronunciation differences Provide pronunciation feedback via bulls eye Simplicity

5. Data Acquisition Data Webcam VIDEO USER Microphone and Webcam to Data Acquisition Data Acquisition Webcam • Auto light compensation • Average Frame Rate • 15 frames per second • Large file stored as a variable • Length of video is short • ~5 Seconds • Camera lights up when recording Data Microphone Microphone • Sampling rate used is adjustable up to 44.1khz • Saved as a variable • Reads in simultaneously with video • Microphone • Built into webcam • Auto noise cancellation • Power comes from computer • Able to crop to only speech Speech USER

6. Repetition of User Data Acquisition Data Play back from variable Allows for a quicker load time Less than 3 seconds to load video Audio and Video do not play in sync Play length ~ 5 Seconds Keeps memory requirements low Window 2 Video of User AUDIO VIDEO USER

7. Diagnostics Can create and graph spectrogram data Allows for determination of vowels using the formants and consonants using the transitions of the formants Can create and graph cepstrum data Inverse Fourier Transform of the log of the Fourier Transform Can find fundamental frequency Can find Zero Crossings Zero Crossings show silence versus speech Data Acquisition Data Window 3 Diagnostics Audio Bulls eye allows for two inputs, along the x and y, graphed as a percent distance from the center Video USER

8. Results Time Domain: Non Native Speaker Time Domain: Native Speaker Spectrogram: Non Native Speaker Spectrogram: Native Speaker

9. Results Continued Cepstrum: Non Native Speaker Cepstrum: Native Speaker Zero : Non Native Speaker Zero Crossings: Native Speaker

10. Design Specifications Goals Read in audio and video at the same time Play back audio and video at the same time within 1 second Implement diagnostics that are sensitive to pronunciation differences Provide pronunciation feedback via bulls eye Simplicity Minimize system requirements Accomplished Can read synchronized audio and video into MATLAB Can play back audio or video separately, or unsynchronized audio and video in MATLAB Can plot diagnostics and find fundamental frequency Can plot on bulls eye All in one webcam as well as keeping the whole program in MATLAB

11. In Progress • Identifying specific components of speech that specific to French • Vowels • Consonants • Quantifying these components and using them on the bulls eye • Creating a GUI • Gather more video samples

12. Future Research • Integrating other languages • Evaluation • Use of non-native speakers • Use of native speakers • Testing in the use of facial communication in oral communication • Basis for comparison of other audible signals

13. Acknowledgements • Professor Rudko • Professor Hanson • Professor Streignitz • Professor Cotter • Professor Catravas • Professor Chilcoat • Professor Pickering • Professor Spallholz