Issues in Spoken Dialogue Systems

1. Issues in Spoken Dialogue Systems Julia Hirschberg LSA07 353

2. Why Spoken Dialogue Systems? • Science • A chance to study and try to emulate human behavior in spoken communication • Technology • Demonstrate utility of several speech technolgies • A means to save human labor and permit flexible management of simple tasks • Online banking services • Reservations (air, train, bus, rental car) • Information access

3. The Instructor • Julia Hirschberg • Columbia Speech Lab

4. Course Overview • Discussion of past and current SDS, with examples of interactions • Major problems in building SDS • Problems in understanding and modeling human conversational behaviors • Limits of the available technologies • Problems in evaluating SDS • SDS of the future

5. Some Early SDS Projects at KTH • The Waxholm Project • Kiosk provides information on travel in the Stockholm archipelago • August • Kiosk provides information on August Strindberg • Other demos at http://www.speech.kth.se/multimodal

6. The Waxholm system Is it possible to eat in Waxholm? I think I want to go to Waxholm Information about the restaurants in Waxholm is shown in this table This is a table of the boats... When do the evening boats depart? The city I want to go tomorrow There are lots of boats from Stockholm to Waxholm on a Friday, At what time do you want to go? Which day of the week do you want to go? I am looking for boats to Waxholm From where do you want to go Thank you Where can I find hotels? Thank you too Information about the hotels in Waxholm is shown in this table Waxholm is shown on this map Information about hotels is shown in this table Which hotels are in Waxholm? Where is Waxholm?

7. Issues • Limited domain: the more so the better • Use of graphical display (maps, tables, charts) to present complex information and enhance visual appeal • Types of initiative in SDS • System • User • Mixed • Importance of lexical priming

8. The August system Yes, it might be that we will! What is your name? Strindberg was born in 1849 What do you do for a living? When were you born? Do you like it here? Good bye! Thank you! You are welcome! Strindberg was married three times! Strindberg was married three times! Over a million people live in the Stockholm area I can answer questions about Strindberg, KTH and Stockholm I call myself Strindberg, but I don’t really have a surname How many people live in Stockholm? People who live in glass houses should not throw stones I come from the department of Speech, Music and Hearing The information is shown on the map Yes, that was a smart thing to say! The Royal Institute of Technology! Perhaps we will meet soon again!

9. Issues • More open domain: entertaining vs. accomplishing a real task • Use of multimodal (speech and face) synthesis • Gesture (face) • System ‘personality’ • A good tool for data collection • How to evaluate?

10. More Recent AT&T Experimental Work • Toot system created to study differences in types of initiative and confirmation strategies and identify user preferences ’98-99 • Some successful dialogues • Some not so successful

11. Issues • Automatic Speech Recognition (ASR) errors • Importance of age, gender, accent • Text-to-Speech (TTS) issues • Error detection and correction • Hyperarticulation • Confirmation strategies • Explicit • Implicit • None • Managing turn-taking

12. Evaluation again….how do we know when a change has been beneficial?

13. A ‘Real’ SDS System from the Lab… • Let’s Go from CMU • What type of initiative supported? • What type of confirmation stragegy? • Difficult or easy to evaluate? • Compare to the AT&T system? The KTH? • How natural is the interaction? • What are they doing to enhance, over the other systems?

14. The Course • Classes, readings, and assignments • Modeling Turn-taking Behaviors • Components of SDS • Automatic Speech Recognition (ASR) • Text-to-Speech (TTS) • Natural Language Understanding (NLU) • Building and Evaluating SDS • Interpreting and Generating Dialogue Acts • Error Detection and Repair Strategies • Making SDS More Human: Entrainment/Adaptation • Can systems become more like their users? • Should they?

15. The Course Paper • Eliza is a classic text-based AI Dialogue System which often fools casual users into thinking they are conversing with a human being. For this assignment: • Become acquainted with the text Eliza: • Identifying 5 or more strategies you can use to demonstrate that Eliza is not a human conversationalist (e.g. linguistic constructions she does not handle well, pragmatic behaviors she does not generate appropriately or recognize). As evidence describe your inputs and her outputs • Next, design a speech-enabled version of Eliza: • Explain the difficulties you will have to overcome in recognition, generation, and dialogue management, compared to the text version, based upon the topics we have studied in class and additional observations you may have • Suggest specific ways of dealing with these problems based upon what you know of the state of the art in SDS or ideas you may have to improve upon it • Indicate the ways in which a spoken version of Eliza might be even better than the text version, in terms of how it might interact with the user and the features of spoken language you might be able to make use of in generation and recognition of user behavior

16. SDS and Popular Culture: SNL • Amtrak’s Julie • Next class: J&M 22.1, Clark ’03, Beattie ‘82