Search and Decoding in Speech Recognition

1. Search and Decoding in Speech Recognition Introduction

2. Introduction • Introduction to Search and Decoding in Speech Recognition: Problem definition: • Communication via Spoken Language • General Spoken Language Understanding System • Concepts and Areas in Speech and Language Understanding • Required Knowledge • Models and Algorithms Veton Këpuska

3. Course overview • Lecture Outline • Assignments • Project • Grading Veton Këpuska

4. Course Outline • Algorithms - Technologies • Introduction • Regular Expressions and Finite State Automata • N-grams • Parts of Speech Tagging • Hidden Markov Models and Automatic Speech Recognition • Context Free Grammars for English Veton Këpuska

5. Course Outline • Parsing – Syntactic and Semantic Analysis • Parsing with Context-Free Grammars • Statistical Parsing • Semantics: Representing Meaning • Computational Semantics • Lexical Semantics • Computational Lexical Semantics Veton Këpuska

6. Course Outline • Applications • Discourse • Information Extraction • Question Answering and Summarization • Machine Translation Veton Këpuska

7. Course Logistics • Lectures: • Two sessions/week, 1.15 hours hours/session • Grading (Tentative) • Assignments 40% • Final Project 60% • Textbook: • Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition.Daniel Jurafsky and James H. MartinPrentice Hall, 2ed • http://www.cs.colorado.edu/~martin/SLP Veton Këpuska

8. Communication via Spoken Language

9. Communication via Spoken Language Output Input Speech Speech Human Computer Text Text Understanding Generation Meaning Veton Këpuska

10. Virtues of Spoken Language Speech interfaces are ideal for information access and management when: • The information space is broad and complex, • The users are technically naive, or • Only telephones are available Veton Këpuska

11. General Spoken Understanding System Lexical Model Language Model Representation Search Veton Këpuska

12. Understanding Natural Understanding Problem

13. Conversational Agent:Space Odyssey 2001 • Dave Bowman: “Open the pod bay doors, HAL” • HAL:“I’am sorry Dave, I’m afraid I can’t do that.” Stanley Kubric and Arthur C. Clarke Screen play of “2001: A Space Oddyssey” Veton Këpuska

14. HAL 9000 Computer • Advanced Computer with following capabilities: • Language Processing • Speaking • Understanding (English) • Lip Reading, etc. Veton Këpuska

15. What it takes to have HAL like computer. • Executive Functions • Understanding Human Input: • Speech Recognition • Natural Language Understanding • Lip Reading • Ability to communicate information comparable to humans: • Natural Language Generation • Speech Synthesis • Information Retrieval: • Finding out where needed textual resources reside • Information Extraction: • Extraction of pertinent Facts from textual resources • Inference • Drawing conclusions based on known facts Veton Këpuska

16. Research Areas • Speech and Language Processing: • Natural Language Processing • Computational Linguistics • Speech Recognition and Synthesis Veton Këpuska

17. Application Areas • Conversational Agents • Dialog Systems • Machine Translation • Question Answering Veton Këpuska

18. Useful Definitions • mor·phol·o·gyPronunciation: mor-'fä-l&-jEFunction: nounEtymology: German Morphologie, from morph- + -logie -logy1 a : a branch of biology that deals with the form and structure of animals and plants b : the form and structure of an organism or any of its parts2 a : a study and description of word formation (as inflection, derivation, and compounding) in language b : the system of word-forming elements and processes in a language3 a : a study of structure or form b : STRUCTURE, FORM4 : the external structure of rocks in relation to the development of erosional forms or topographic features Veton Këpuska

19. Useful Definitions • pho·nol·o·gyPronunciation: f&-'nä-l&-jE, fO-Function: nounDate: 17991 : the science of speech sounds including especially the history and theory of sound changes in a language or in two or more related languages2 : the phonetics and phonemics of a language at a particular time • pho·net·icsPronunciation: f&-'ne-tiksFunction: noun plural but singular in constructionDate: 18361 : the system of speech sounds of a language or group of languages2 a : the study and systematic classification of the sounds made in spoken utterance b : the practical application of this science to language study Veton Këpuska

20. Useful Definitions • pho·neme Pronunciation: fō-ˌnēm Function: noun : any of the abstract units of the phonetic system of a language that correspond to a set of similar speech sounds (as the velar \k\ of cool and the palatal \k\ of keel) which are perceived to be a single distinctive sound in the language • phonemicsPronunciation: f&-'ne-miksFunction: noun plural but singular in constructionDate: 19361: a branch of linguistic analysis involving the study of phonemes 2: the structure of a language in terms of phonemes • pho·no·tac·ticsPronunciation: fo-n&-'tak-tiksFunction: noun plural but singular in constructionDate: 1956: the area of phonology concerned with the analysis and description of the permitted sound sequences of a language Veton Këpuska

21. Example: Veton Këpuska

22. Cool: Veton Këpuska

23. Keel: Veton Këpuska

24. Knowledge in Speech and Language Processing

25. Knowledge in Speech & Language Processing • Techniques that process Spoken and Written human language. • Necessary use of knowledge of language. • Example: Unix wc command: • Counts bytes and number of lines that a text file contains. • Also counts number of words contained in a file. • Requires knowledge of what it means to be a word. Veton Këpuska

26. Example • http://www.bing.com/videos/search?q=2001+a+space+odyssey+dave+bowman&form=VIRE1&first=1#view=detail&mid=99B0950DDBF6B1C3EE5599B0950DDBF6B1C3EE55 Veton Këpuska

27. Knowledge in Speech & Language Processing • HAL ⇦ David: • Requires analysis of audio signal: • Generation of exact sequence of the words that David is saying. • Analysis of additional information that determines meaning of that sequence of the words. • HAL ⇨ David • Requires ability to generate an audio signal that can be recognized: • Phonetics, • Phonology, • Synthesis, and • Syntax (English) • Morphology Veton Këpuska

28. Knowledge in Speech & Language Processing • Hal must have knowledge of morphology in order to capture the information about the shape and behavior of words in context: • Semantics Veton Këpuska

29. Knowledge in Speech & Language Processing • Beyond individual words: • HAL must know how to analyze the structure of Dave’s utterance. • REQUEST: HAL, open the pod bay door • STATEMENT: HAL, the pod bay door is open • QUESTION: HAL, is the pod bay door open? • HAL must use similar structural knowledge to properly string together the words that constitute its response (Syntax): • I’m I do, sorry that afraid Dave I’m can’t. • (I’m sorry Dave, I’m afraid I can’t do that.) Veton Këpuska

30. Knowledge in Speech & Language Processing • Knowing the words and Syntactic structure of what Dave said does not tell HAL much about the nature of his request. • Knowledge of the meanings of the component words is required (lexical semantics) • Knowledge of how these components combine to form larger meanings (compositional semantics). Veton Këpuska

31. Knowledge in Speech & Language Processing • Despite its bad behavior HAL knows enough to be polite to Dave (pragmatics). • Direct Approach: • No • No, I won’t open the door. • Embellishment: • I’m sorry • I’m afraid • Indirect Refusal: I can’t • Direct Refusal: I won’t. Veton Këpuska

32. Knowledge in Speech & Language Processing • Instead simply ignoring Dave’s request HAL chooses to engage in a structured conversation relevant to Dave’s initial request. • HAL’s correct use of the words “that” in its answer to Dave’s request is a simple illustration of the kind of between-utterance device common in such conversations. • Correctly structuring such conversations requires knowledge of discourse conventions. Veton Këpuska

33. Knowledge in Speech & Language Processing • In the following question: • How many states were in the United States that year? • One needs to know what “that year” refers too. • Coreference Resolution Veton Këpuska

34. Summary • Phonetics and Phonology: • The study of linguistic sounds • Morphology: • The study of the meaningful components of words. • Syntax: • The study of the structural relationships between words. • Semantics: • The study of meaning • Pragmatics: • The study of how language is used to accomplish goals. • Discourse: • The study of linguistic units larger then a single utterance. Veton Këpuska

35. Ambiguity Veton Këpuska

36. Ambiguity • Most if not all tasks in speech and language processing can be viewed as resolving ambiguity. • Example: • I made her duck. Veton Këpuska

37. Ambiguity • I made her duck. • Possible interpretations: • I cooked waterfowl (e.g., duck) for her • I cooked waterfowl (e.g., duck) belonging to her • I created (plaster?) duck she owns. • I caused her to quickly lower her head or body. • I waived my magic want and turned her into waterfowl (e.g., duck). Veton Këpuska

38. Ambiguity These different meanings are caused by a number of ambiguities. • First, the words duckand herare morphologically or syntactically ambiguous in their part-of-speech. • Duck can be a verb or a noun, while • her can be a dative pronoun or a possessive pronoun. Veton Këpuska

39. Duck (webster.com) 1duck noun,often attributive \ˈdək\, plural ducks 1or plural duck a : any of various swimming birds (family Anatidae, the duck family) in which the neck and legs are short, the feet typically webbed, the bill often broad and flat, and the sexes usually different from each other in plumage b : the flesh of any of these birds used as food 2 : a female duck — compare drake 3 chiefly British : darling —often used in plural but singular in construction 4 :person, creature intransitive verb 1 a: to plunge under the surface of water b: to descend suddenly :dip 2 a: to lower the head or body suddenly :dodge b:bow, bob 3 a: to move quickly b: to evade a duty, question, or responsibility duckverb Definition of DUCK transitive verb 1 : to thrust under water 2 : to lower (as the head) quickly :bow 3 :avoid, evade <duck the issue> Veton Këpuska

40. Her (webster.com) 1heradj \(h)ər, ˈhər\ Definition of HER : of or relating to her or herself especially as possessor, agent, or object of an action <her house> <her research>  2herpronoun objective form ofshe dative pronoun — used to refer to a certain woman, girl, or female animal as the object of a verb or a preposition ▪ Tell her I said hello. ▪ Did you invite her? ▪ I gave the book to her. ▪ a gift for her ▪ The dress fits her sister as well as her possessive pronoun I gave her book back to her. Veton Këpuska

41. Ambiguity • Second, the word makeis semantically ambiguous; it can mean create or cook. • Finally, the verb make is syntactically ambiguous in a different way. • Make can be transitive, that is, taking a single direct object (2), or • it can be ditransitive, that is, taking two objects (5), meaning that the first object (her) got made into the second object (duck). • Finally, make can take a direct object and a verb (14), meaning that the object (her) got caused to perform the verbal action (duck). Veton Këpuska

42. Make (dictionary.com) 1 make [meyk]  Show IPAverb, made, mak·ing, noun verb (used with object) 1.to bring into existence by shaping or changing material, combining parts, etc.: to make a dress; to make a channel; to make a work of art. 2.to produce; cause to exist or happen; bring about: to make trouble; to make war. 3.to cause to be or become; render: to make someone happy. 4.to appoint or name: The President made her his special envoy. 5.to put in the proper condition or state, as for use; fix; prepare: to make a bed; to make dinner. 6.to bring into a certain form: to make bricks out of clay. 7.to convert from one state, condition, category, etc., to another: to make a virtue of one's vices. 8.to cause, induce, or compel: to make a horse jump a barrier. 9.to give rise to; occasion: It's not worth making a fuss over such a trifle. Veton Këpuska

43. Make (dictionary.com) 10.to produce, earn, or win for oneself: to make a good salary; to make one's fortune in oil. 11.to write or compose: to make a short poem for the occasion. 12.to draw up, as a legal document; draft: to make a will. 13.to do; effect: to make a bargain. 14.to establish or enact; put into existence: to make laws. 15.to become by development; prove to be: You'll make a good lawyer. 16.to form in the mind, as a judgment or estimate: to make a decision. 17.to judge or interpret, as to the truth, nature, meaning, etc.(often followed by of ): What do you make of it? 18.to estimate; reckon: to make the distance at ten miles. 19.to bring together separate parts so as to produce a whole; compose; form: to make a matched set. 20.to amount to; bring up the total to: Two plus two makes four. That makes an even dozen. 21.to serve as: to make good reading. Veton Këpuska

44. Make (dictionary.com) 22.to be sufficient to constitute: One story does not make a writer. 23.to be adequate or suitable for: This wool will make a warm sweater. 24.to assure the success or fortune of: a deal that could make or break him; Seeing her made my day. 25.to deliver, utter, or put forth: to make a stirring speech. 26.to go or travel at a particular speed: to make 60 miles an hour. 27.to arrive at or reach; attain: The ship made port on Friday. Do you think he'll make 80? 28.to arrive in time for: to make the first show. 29.to arrive in time to be a passenger on (a plane, boat, bus, train, etc.): If you hurry, you can make the next flight. 30.Informal . to gain or acquire a position within: He made the big time. 31.to receive mention or appear in or on: The robbery made the front page. 32.to gain recognition or honor by winning a place or being chosen for inclusion in or on: The novel made the best seller list. He made the all-American team three years in a row. Veton Këpuska

45. Make (dictionary.com) 33.Slang . to have sexual intercourse with. 34.Cards . a. to name (the trump). b. to take a trick with (a card). c. Bridge . to fulfill or achieve (a contract or bid). d. to shuffle (the cards). 35.to earn, as a score: The team made 40 points in the first half. 36.Slang . (especially in police and underworld use) a. to recognize or identify: Any cop in town will make you as soon as you walk down the street. b. to charge or cause to be charged with a crime: The police expect to make a couple of suspects soon. 37.to close (an electric circuit). 38.South Midland and Southern U.S. to plant and cultivate or produce (a crop): He makes some of the best corn in the country. Veton Këpuska

46. Make (dictionary.com) verb (used without object) 39.to cause oneself, or something understood, to be as specified: to make sure. 40.to show oneself to be or seem in action or behavior (usually followed by an adjective): to make merry. 41.to be made, as specified: This fabric makes up into beautiful drapes. 42.to move or proceed in a particular direction: They made after the thief. 43.to rise, as the tide or water in a ship. 44.South Midland and Southern U.S. (of a crop) to grow, develop, or mature: It looks like the corn's going to make pretty good this year. 45.make down, Chiefly Pennsylvania German . to rain or snow: It's making down hard. 46.make fast, Chiefly Nautical . to fasten or secure. 47.make shut, Chiefly Pennsylvania German . to close: Make the door shut. Veton Këpuska

47. Make (dictionary.com) Noun 48.the style or manner in which something is made; form; build. 49.production with reference to the maker;  brand: our own make. 50.disposition; character; nature. 51.the act or process of making. 52.quantity made; output. 53.Cards . the act of naming the trump, or the suit named as trump. 54.Electricity . the closing of an electric circuit. 55.Jewelry . the excellence of a polished diamond with regard to proportion, symmetry, and finish. 56.Slang . identifying information about a person or thing from police records: He radioed headquarters for a make on the car's license plate. Veton Këpuska

48. Approaches for Disambiguation • In this class you will be introduced: • The models and algorithms as ways to resolve or disambiguate these ambiguities. For example: • deciding whether duckis a verb or a noun can be solved by part-of-speech tagging. • Deciding whether makemeans “create” or “cook” can be solved by word sense disambiguation. • Resolution of part-of-speech and word sense ambiguities are two important kinds of lexical disambiguation. A wide variety of tasks can be framed as lexical disambiguation problem. For example, • A text-to-speech synthesis system reading the word leadneeds to decide whether it should be pronounced as in lead pipe or as in lead me on. • By contrast, deciding whether • her and duck are part of the same entity (as in (1) or (4)) or • are different entity (as in (2)) is an example of syntactic disambiguation and can be addressed by probabilistic parsing. • Ambiguities that don’t arise in this particular example (like whether a given sentence is a statement or a question) will also be resolved by speech act interpretation. Veton Këpuska

49. Models and Algorithms Veton Këpuska

50. Models and Algorithms • One of the key insights of the last 50 years of research in language processing is that the various kinds of knowledge described in the last sections can be captured through the use of a small number of formal models, or theories. • Fortunately, these models and theories are all drawn from the standard toolkits of computer science, mathematics, and linguistics and should be generally familiar to those trained in those fields. • Among the most important models are • State machines, • Rule systems, • Logic, • Probabilistic models, and • Vector-space models. • These models, in turn, lend themselves to a small number of algorithms, among the most important of which are • state space search algorithms such as dynamic programming, and • machine learning algorithms such as classifiersand EM (Estimation Maximization) and other learning algorithms Veton Këpuska