LOT summer school Ultrasound, phonetics, phonology: Articulation for Beginners!

1. LOT summer schoolUltrasound, phonetics, phonology: Articulation for Beginners! With special thanks to collaborators Jane Stuart-Smith & Eleanor Lawson Joanne Cleland & Zoe Roxburgh Natasha Zharkova, Laura Black, Steve Cowen ReenuPunnoose, KoenSebreghts Sonja Schaeffler & Ineke Mennen ConnyHeyde Alan Wrench (aka Articulate Instruments Ltd) for AAA software and UTI hardware Various funding – thank you to ESRC, EPSRC, QMU June 2013 James M Scobbie CASL Research Centre

2. Introduction to articulation • Brief overview of techniques • Ultrasound tongue imaging • Playtime • Technical issues and the nitty gritty of data • Maybe a linguistic illustration • Malayalam liquids Structure

3. Why study articulation?

4. It underlies acoustic and visual elements of speech, but is only a means to an end • It tells you about what speakers actually do • It might be what the speaker intends to control • Some bits of speech are silent • It is interesting in its own right because it is a complex multichannel linguistic phenomenon • Applications – clinical, military, HCI, L2 • Comparison with sign language and gesture • Provides evidence for phonological patterns Why study articulation?

5. Silent articulations • Pre-speech, post-speech • Speech errors • Voiceless stops • Listening, turn taking • Covert contrasts • In acquisition therapy, L2 learning, sociolinguistics • Covert errors • In acquisition, therapy, L2 learning • Articulation / acoustics relationship in segmental and prosodic speech Easy topics for research

6. Techniques for speech analysis?

7. Flesh-point tracking • EMA Electromagnetic Articulography • Motion-capture • Constriction tracking • EPG Electropalatograph • EGG / Largyngograph & transglottal illumination • Parameter tracking / indirect analysis • Airflow • Intraoral pressure sensing • EMG Electromyography or muscular measures • Acoustic analysis • formants, pitch, voice quality, constriction types, VOT etc. Quantitative articulatory approaches

8. Video (including basic photos/still frames) • Regular video (25fps PAL or 30fps NTSC), often underlyingly a higher image / refresh rate (de-interlaceable) • From cheap camcorders to endoscopy for internal images • X-ray stills and X-ray cinematography • MRI Magnetic Resonance Imaging & CT scanning • Excellent resolution of superficial and deep features in 3D for static images • Bigger voxels, more grainy, more processing at faster frame rates, great prospects in next few years • UTI Ultrasound Tongue Imaging • Regular video outputs or • Digital (“high speed”) cineloop outputs Imaging

9. EMA: gluing and sitting in the cube: 2h+ data • UTI: probe fitting and the headset: “30m” data • Short sessions, outputs are image sequences, needs synchronisation, captures root What hi-tech speakers go though

10. Video playback in powerpoint / media players don’t really convey the spatio-temoral nature of the data Vid UTI: ECB08 spontaneous dialogueMRI: USC

11. What do you do when you “say hello”…? • Description…? What about something easier?

12. Same speaker? MRI

13. What do you do when you “say hello”…? Video can tell you a lot

14. Easy-to-get images are only 2 dimensional • The head and vocal tract are 3D objects • Which 2 dimensional plane do you want to study? • Speaker & camera move relative to each other • False motion of articulators within the plane • Towards or away from the camera, changing scale • And rotations mean a different plane is shown • Not many frames per second • Potential for smearing in time • Missing key events completely • Weak and/or variable synch with acoustics Drawbacks are not unique

15. To get data in more than one plane, let alone enough to make a 3D image that moves in time… … means sacrifices • Lower spatial resolution • Lower temporal resolution 3D / 4D?

16. Ultrasound Tongue Imaging

17. Ultrasound as a tongue imaging technique • Relatively cheap, non-invasive and accessible • Fieldwork • Clinical diagnostics • Child language acquisition • Standard laboratory phonetics & phonology • Real time visual biofeedback • Phonetics and linguistics teaching • Clinical intervention • L2 teaching and personal training UTI applications

18. Quick, portable, cheap, live/realtime, “comfyish” • Synchronisation with audio, probe movement • Applications • Clinic • Teaching • Piloting • Outreach • Fieldwork • Discourse • Infants • pT Research! hand held & live video-mode

19. Articulate Assistant Advanced (AAA) • ~120fps hs-UTI: raw probe echo-location data is stored and re-imaged on the fly • Up to ~400fps available • 135° Field of View • ~60fps de-interlaced lip camerastored as uncompressed bitmap QMU AAA multichannel lab set-up

20. MC suburb and the AAA multichannel system

21. Data collection and analysis in a fast single dedicated software environment • Ultrasonix high speed UTI (no post-processing) • Various video UTI or camcorder systems • Same annotation & display software for EPG, EMA • Custom-made multichannel synchronisation • Video via “synch-brightup” clapperboard on AD of video images, with built in batch-processing • De-interlacing from ~30 to ~60-fps • Offset of clapper-frame to adjust for UTI creation (~20ms) • Semi-automatic edge-detection • Smoothing & confidence-rating over 42 points AAA

22. Ultrasound gives rise to • Artefacts from parallel tongue & echo pulse beams • Missing data • Between scan lines, or beyond the scan area • Behind bones • Above a sublingual cavity (aka losing the tip) • Grainy data or poor resolution • Older speakers, dry mouth, beards, etc. • Tongue surface when it’s far from the probe • When tongue is parallel to the scanlines • Problems with stabilisation & synchronisation • Technical solutions can lead to speaker fatigue Some problems

23. We only have mid-sagittal tongue curves • Not passive articulators • Not all the tongue surface • Not all the internal tongue tissue • Not lips • But unlike EMA • We are not limited to 3 or 4 anterior points • And unlike MRI • UTI is cheap, non-invasive, portable and quick • For small datasets analysis is quick… we can collect & trace 12 tokens of 5 vowels in half a day With UTI…

24. Real time visual biofeedback • Have a go… • Front/back vowels • Dutch & English /r/ • Dutch & English /s/ and English “sh” • Swallowing Annotation and analysis software • AAA • Basic overview, using some good data • Annotation and filtering • Drawing some splines Playtime!