Curs 09

1. Curs 09 Speech Cristian Miron

2. Summary • Why to use Speech in a car? • Speech synthesis • Speech recognition

3. Why Speech? • Difficulties on using touch screen or the buttons from board • Using Speech, the drivers' eyes are always on the road (give commands to machine, hear announcements from navigation, etc.) • What can be done with speech in a car: • Control media player / radio stations • Control navigation • Control phone book and phone activities • Find POI locations (POI : point of interests) • Get machine status • Customizations: change language, voice

4. Speech overview • Difficulties with speech, scenarios and solutions: • if the driver listens music or radio, how speech recognition will work? (barge-in) • same problem, when using TTS (barge-in) • multiple persons speaking: "personal space" ideas • Type of grammars for recognition: • static (for command and control) • text enrollment (for dynamic data like media player artists, songs names, etc.) • voice tags (for dynamic data like radio stations names, persons from phone book)

5. Example of SUI (Speech User Interface) state diagram

6. Speech synthesis • What is speech synthesis? • History • How it works? • Synthesizer technologies • Challenges • Voice synthesizers • Applications

7. What is speech synthesis? • Speech synthesis is the artificial production of human speech. • Can be implemented in software or hardware • A text-to-speech (TTS) system converts normal language text into speech • Synthesized speech can be created by concatenating pieces of recorded speech that are stored in a database

8. What is speech synthesis? • The quality of a speech synthesizer is judged by its similarity to the human voice (naturalness), and by its ability to be understood (intelligibility). • Naturalness describes how closely the output sounds like human speech, while intelligibility is the ease with which the output is understood • An intelligible text-to-speech program allows people with visual impairments or reading disabilities to listen to written works on a home computer • The ideal speech synthesizer is both natural and intelligible

9. History • Sec. X-XII: Long before electronic signal processing was invented, there were those who tried to build machines to create human speech. (early legends of the existence of "speaking heads“) • 1779: the Danish scientist Christian Kratzenstein, working at the Russian Academy of Sciences, built models of the human vocal tract that could produce the five long vowel sounds • 1930: Bell Labs developed the VOCODER, a keyboard-operated electronic speech analyzer and synthesizer that was said to be clearly intelligible • 1968: first complete text-to-speech system • Early electronic speech synthesizers sounded robotic and were often barely intelligible. However, the quality of synthesized speech has steadily improved, and output from contemporary speech synthesis systems is sometimes indistinguishable from actual human speech.

10. How it works? • The two primary technologies for generating synthetic speech waveforms are concatenative synthesis and formant synthesis • A text-to-speech system (or "engine") is composed of four parts: a front-end and a back-end • The front-end has two major tasks: • text normalization: it converts raw text containing symbols like numbers and abbreviations into the equivalent of written-out words • grapheme-to-phoneme conversion: it assigns phonetic transcriptions to each word, and divides and marks the text into prosodic units, like phrases, clauses and sentences • The back-end (synthesizer): converts the symbolic linguistic representation into sound.

11. Overview of a typical TTS system

12. Synthesizer technologies • Concatenative synthesis (based on the concatenation of segments of recorded speech): • Unit selection synthesis • Diphone synthesis • Domain-specific synthesis • Formant synthesis (does not use human speech samples at runtime) • Articulatory synthesis (refers to computational techniques for synthesizing speech based on models of the human vocal tract) • HMM-based synthesis (based on hidden Markov models)

13. Concatenative synthesis • Generally, concatenative synthesis produces the most natural-sounding synthesized speech • Unit selection synthesis: • uses large databases of recorded speech • provides the greatest naturalness, because it applies only a small amount of digital signal processing (DSP) to the recorded speech • maximum naturalness typically requires unit-selection speech databases to be very large, in some systems ranging into the gigabytes of recorded data, representing dozens of hours of speech • Diphone synthesis: • uses a minimal speech database containing all the diphones (sound-to-sound transitions) occurring in a language • the number of diphones depends on the phonotactics of the language: for example, Spanish has about 800 diphones, and German about 2500 • the quality of the resulting speech is generally worse than that of unit-selection systems, but more natural-sounding than the output of formant synthesizers

14. Concatenative synthesis • Domain-specific synthesis • concatenates prerecorded words and phrases to create complete utterances • It is used in applications where the variety of texts the system will output is limited to a particular domain, like transit schedule announcements or weather reports • Formant synthesis • does not use human speech samples at runtime • the synthesized speech output is created using an acoustic model • parameters such as fundamental frequency, voicing and noise levels are varied over time to create a waveform of artificial speech

15. Challenges • Text normalization challenges • Text-to-phoneme challenges • Evaluation challenges • Prosodics and emotional content

16. Text normalization challenges • Texts are full of heteronyms, number and abbreviations that all require expansion into a phonetic representation • Most text-to-speech (TTS) systems do not generate semantic representations of their input texts • Various heuristic techniques are used to guess the proper way to disambiguate homographs, like examining neighboring words and using statistics about frequency of occurrence. • Deciding how to convert numbers is another problem that TTS systems have to address • Abbreviations can be ambiguous

17. Text-to-phoneme challenges • dictionary-based approach: • a large dictionary containing all the words of a language and their correct pronunciations is stored by the program • determining the correct pronunciation of each word is a matter of looking up each word in the dictionary and replacing the spelling with the pronunciation specified in the dictionary • it's quick and accurate, but completely fails if it is given a word which is not in its dictionary • rule-based approach: • pronunciation rules are applied to words to determine their pronunciations based on their spellings • works on any input, but the complexity of the rules grows substantially as the system takes into account irregular spellings or pronunciations

18. Evaluation challenges • The consistent evaluation of speech synthesis systems may be difficult because of a lack of universally agreed objective evaluation criteria • The quality of speech synthesis systems also depends to a large degree on the quality of the production technique and on the facilities used to replay the speech

19. Voice synthesizers and API’s • Microsoft Speech API • SoftVoice TTS • SVOX TTS • Apple PlainTalk • FreeTTS • Nuance Vocalizer • Vocoder • eSpeak

20. Applications • Accessibility (speech synthesis has long been a vital assistive technology tool and its application in this area is significant and widespread) • News service (some news sites used speech synthesis to convert written news to audio content, which can be used for mobile applications) • Entertainment (Speech synthesis techniques are used as well in the entertainment productions such as games, animations) • Navigation

21. Speech recognition • Agenda • What is speech recognition? • History • How it works? • Voice synthesizers • Applications • Challenges

22. What is speech recognition? • Speech recognition converts spoken words to machine-readable input

23. What is speech recognition? • The performance of speech recognition systems is usually specified in terms of accuracy and speed • Accuracy may be measured in terms of performance accuracy which is usually rated with word error rate (WER) • Speed is measured with the real time factor (If it takes time P to process an input of duration I, the real time factor is defined as RTF = P / I) • Optimal conditions usually assume that users: • have speech characteristics which match the training data, • can achieve proper speaker adaptation, and • work in a clean noise environment (e.g. quiet office or laboratory space). • Limited vocabulary systems, requiring no training, can recognize a small number of words (for instance, the ten digits) as spoken by most speakers

24. History • 1984 – First system of speech recognition based on a supercomputer. Each computing process lasts few minutes. Only 5000 words from English are recognized. • 1986 - Tangora 4 Prototype: Using a special microprocessor, real-time computing of a human voice on a desktop PC. • 1993 - IBM Personal Dictation: first solution for PC - 1000 USD. • 1998 - IBM, Dragon, Lernout&Hauspie si Philips launch first customer versions of their products

25. How is works? • Modern statistically-based speech recognition algorithms: • Acoustic model • Language model • Hidden Markov model (HMM)-based speech recognition (widely used in many systems) • Dynamic time warping (DTW)-based speech recognition

26. Hidden Markov model (HMM)-based speech recognition • Modern general-purpose speech recognition systems are generally based on Hidden Markov Models • These are statistical models which output a sequence of symbols or quantities • One possible reason why HMMs are used in speech recognition is that a speech signal could be viewed as a piecewise stationary signal or a short-time stationary signal • Another reason why HMMs are popular is because they can be trained automatically and are simple and computationally feasible to use • In speech recognition, the hidden Markov model would output a sequence of n-dimensional real-valued vectors (with n being a small integer, such as 10), outputting one of these every 10 milliseconds

27. Dynamic time warping (DTW)-based speech recognition • Dynamic time warping is an approach that was historically used for speech recognition but has now largely been displaced by the more successful HMM-based approach • Dynamic time warping is an algorithm for measuring similarity between two sequences which may vary in time or speed. • For instance, similarities in walking patterns would be detected, even if in one video the person was walking slowly and if in another they were walking more quickly, or even if there were accelerations and decelerations during the course of one observation

28. Applications • Health care • Military • High-performance fighter aircraft • Helicopters • Battle management • Training air traffic controllers • Telephony and other domains • People with disabilities • Mobile telephony, including mobile email • Robotics • Video games • Home automation • Automotive speech recognition • Hands-free computing: voice command recognition computer user interface

29. Speech Recognition engines • Microsoft Speech Server • IBM: WebSphere Voice Server • Nuance: VoCon • VoiceBox

30. Speech Recognition in Linux • There is currently no open-source equivalent of proprietary speech recognition software for Linux • However, there are several incomplete, open-source projects and solutions that could be used to attain some elements of speech recognition in the free operating system: • CVoiceControl • DynaSpeak • VoxForge • Julius

31. Challenges • Variability of Speech Patterns • Processing Power • Extracting Meaning • Background Noise • Continuous Speech Recognition

32. Challenges of speech recognition in automotive industry • Noisy environment, the noise level of the signal is between 20dB and 5db • Cheap microphones (low cost) placed 30-100 cm from speaker • Embedded platforms with restrictions related to CPU and memory/space storage

33. Speech recognition in automotive industry • Ford SYNC (developed by Ford and Microsoft) • Continental MMP (developed by Continental/Siemens)

34. Speech Application Programming Interface (SAPI) • The Speech Application Programming Interface or SAPI is an API developed by Microsoft to allow the use of speech recognition and speech synthesis within Windows applications . • In general all versions of the API have been designed such that a software developer can write an application to perform speech recognition and synthesis by using a standard set of interfaces, accessible from a variety of programming languages. • In addition, it is possible for a 3rd-party company to produce their own Speech Recognition and TTS engines or adapt existing engines to work with SAPI.

35. SAPI 5 API • Features included: • Shared recognizer • In-proc recognizer • Grammar objects • Voice object • Audio interfaces • User lexicon object • Object tokens

36. Nice things • Faster chip-based speech recognition will enable video players to search rapidly for Arnold Schwarzenegger saying "Hasta la vista, baby!" in the movie • And lower power consumption will enable a cell phone to take dictated notes.

37. Voice Recognition Funny Errors • While typing an operating report: • "The patient was prepped and raped in the usual fashion" instead of "The patient was prepped and draped in the usual fashion." • "she is buried" instead of "she is married"

38. Speech • Questions?

39. References • Speech synthesis - http://en.wikipedia.org/wiki/Speech_synthesis • Speech recognition - http://en.wikipedia.org/wiki/Speech_recognition • SAPI - http://en.wikipedia.org/wiki/Speech_Application_Programming_Interface