Interactive Dialogue for Simulation with Virtual Characters

1. Interactive Dialoguefor Simulation with Virtual Characters David Traum traum @ ict.usc.edu http://www.ict.usc.edu/~traum

2. ICT Dialogue Group - Fall 2006 • Staff: Dr David Traum (CS) Dr Anton Leuski (CS), Dr Bilyana Martinovski (Ling),Dr Mark Core (CS), Susan Robinson (Ling), Jillian Gerten (Ling) • CS Students: Antonio Roque, Sudeep Gandhe, Dusan Jan, Ashish Vaswani • Temporary visitors and Interns • Close collaboration with other groups at ICT, ISI, Campus, others

3. Dialogue: Terms • Dialogue • Communication involving: • Multiple contributions • Coherent Interaction • More than one participant • Dialogue System • System that engages in dialogue with users and/or other systems • Dialogue Model • Formal characterization of dialogue, evolving context, and possible/likely continuations • Dialogue Manager • Module of a system concerned with dialogue modelling and decisions of how to contribute to dialogue

4. Dialogue Management Tasks • Maintaining & Updating Context • Deciding what to say next • Interface with non-communicative reasoning aspects of agent • Provide expectations for interpretation

5. Dialogue Research • Study of human conversational behavior • Pragmatic meaning • Representation and use of context • Understanding and Generation of communicative behavior • Interactional patterns • Methods • Empirical investigation • Theory development • Construction of dialogue systems • Empirical evaluation • Construction of conversational agents • Ability to converse and act in a specific domain • Integration with other abilities (perception, cognition, emotion)

6. Dialogue Model • Representation of context of interaction including • Agent’s internal state • World state • State of others • Social relationships • interaction • Model of context-changing actions • Observation of communication • Performance of communication • Other observations • Thinking/reasoning

7. Information-State Approach to Computational Dialogue Modeling (Larsson& Traum 2000, Traum &Larsson 2003) • Information Specification • Dialogue Moves for Update • Behaviors Dialogue moves • Toolkits for easy implementation (TrindiKit, dipper, Midiki, USC Steve/Austin) • Modular approach to sub-components • More direct relation to theory • Modular approach to theory

8. Dialogue Acts Dialogue Acts Dialogue Approach:Layered Information State • Layer captures coherent aspect of communicative interaction (e.g., turn, grounding, obligations) • Layer consists of • Information State components (state of interaction) • Dialogue Acts (Packages of changes to information state) Input Utterance Recognition Rules Update Rules Info State Components Selection Rules Output Utterance (verbal and nonverbal) Realization Rules Dialogue Manager

9. Methodology Implement on the basis of model Build models rules Model Study Build Acquire data Test Discover gaps in data Design evaluation

10. Recent ICT Dialogue Agents • Embodied • Question-answering characters • Be a reporter • Sgt Blackwell • Tactical Questioning for Army • Mediation for interviews (AXL) • Virtual Humans for complex role-play interaction • MRE - characters for multi-party collaboration • SASO - non-team negotiation • Group conversation • Non-embodied • Transonics Farsi-English spoken translation for medical interviews • Radiobots - military radio operators for training in VR simulation

11. Example 1: Sgt Blackwell • Focus: technology demo • Highlights: • Life-sized, mixed reality • Trans-screen • High-production quality • Rendering (> 60K polygons) • Voice • Authored Text • Robust responsiveness • Speech recognition and speech and non-verbal reply • Limited domain of interaction: responding to interview/Q&A

12. Sgt Blackwell Video

13. Sgt Blackwell “Dialogue Model” • Set of pre-constructed answers • In domain • Off-topic • Prompt • Local history • IR-based classification • Given possibly unseen question, map to best answer

14. MRE SASO-ST ICT Virtual Human Domains

15. Immersive Training Environment • Mission Rehearsal Exercise(Swartout et al ‘01) • Human lieutenant (student) faces peacekeeping dilemmas • Appears in video offsceen • Artificial agents interact with user • Mentor (e.g., sergeant, front left) • Teammates (e.g., medic, front right) • Locals (e.g., mother, front center) • VR Theatre • 8’ 150˚ Curved Screen, • Multiple Projectors • 10-2 3-d spatialized sound

16. Communication Contact Attention Conversation Participants Turn Initiative Grounding Purpose Rhetorical Social Obligations-Commitments Negotiation-Collaboration Social Roles Individual Perception Rational belief,desire, intention,.. Emotional Coping strategies MRE Dialogue Layers (Traum & Rickel AAMAS 2002)

17. Grounding(Clark & Schaeffer 89, Traum 94) • Signal by B of how s is interpreted • Possibly revised signals by A • Grounded state = mutually believed common ground

18. MRE Team-Negotiation Example

19. Secure Area Secure 12-4 Secure 8-12 Render Aid Secure 4-8 Secure LZ Secure Accident Medevac Sgt’s Negotiation Behavior 1 Focus=1 Lt: U9 “secure a landing zone” Committed(lt,7,sgt), 7 authorized, Obl(sgt,U9) Sgt: U10 “first we should secure the assembly area” Disparaged(sgt, 7,lt), endorsed(sgt,2.lt), grounded(U9) Lt: U11“secure the area” Committed(lt,2,sgt), 2 authorized, Obl(sgt,U11),grounded(U10) Sgt: U12“yes sir” Committed(sgt,2,lt), grounded(U11), Push(2,focus) Goal7:Announce(2,{1sldr,2sldr,3sldr,4sldr}) Goal8: Start-conversation(sgt, ,{1sldr,2sldr,…},2) Goal8 ->Sgt: U21 “Squad leaders listen up!” Goal7 ->Sgt:U22 “I want 360 degree security” Committed(sgt,2,{1sldr,2sldr,3sldr,4sldr}) Push(3, focus) Goal9:authorize 3 Goal9 ->Sgt:U23“1st squad take 12-4” Committed(sgt,3, {1sldr,2sldr,3sldr,4sldr}), 3 authorized Pop(3), Push(4) Goal10: authorize 4 Goal10 ->Sgt: U24“2nd squad take 4-8” Committed(sgt,4,{1sldr,2sldr,3sldr,4sldr}), 4 authorized Pop(4) … A10: Squads move Grounded(U21-U26) ends conversation about 2, Happened(2) Push(7,Focus) Decomposition Area Secure Squads in area A=Lt, R=Sgt A=Lt ,R=S 2 7 Decomposition 3 4 A=Sgt,R=1sldr A=Sgt,R=2sldr 5 6 A=Sgt,R=3sldr A=Sgt,R=4sldr

20. SASO-ST: Beyond Team negotiation

21. SASO-ST video

22. Negotiation Strategies (Traum et al IVA 2005) • Result from orientations toward negotiation: • Avoidance • avoid • Distributive • attack • Integrative • negotiate • Govern choice of dialogue move, posture, and interpretation

23. Current Research Topics • Dialogue genre taxonomy • How is language use similar and different depending on aspects of Activity (participants, culture, goals, roles, setting,..) • Multi-party, multi-modal dialogue • Algorithms for dialogue processing • Multiple approaches (theory and data-driven) • Strategies for dialogue management • Parameterizable dialogue models • Personality models • Culture-specific aspects of production and interpretation • Affective aspects of dialogue • Immersion/engagement • Effects of emotion/coping on processing • Emotion-oriented language (e.g., empathy, mitigation, boosting)

24. Opportunities • Come talk to our agents at ICT • Contact gerten@ict.usc.edu • ICT Summer internship program • Apply on ICT webpage www.ict.usc.edu (specify areas of interest) • Possibilities for directed research • Inquire traum@ict.usc.edu