1 / 64

C onversational A gents in M ulti- P arty I nteractive S ituations

C onversational A gents in M ulti- P arty I nteractive S ituations. Rohit Kumar. Committee Carolyn P. Rosé (Chair) Alan W. Black Ian R. Lane Jason D. Williams (AT&T Research). Friday, June 11, 2010. Bridges. Multi-Party Interaction. Social / Trust Games. Collaborative Learning.

june
Download Presentation

C onversational A gents in M ulti- P arty I nteractive S ituations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Conversational Agents in Multi-Party Interactive Situations Rohit Kumar Committee Carolyn P. Rosé (Chair) Alan W. Black Ian R. Lane Jason D. Williams (AT&T Research) Friday, June 11, 2010

  2. Bridges Multi-PartyInteraction Social /Trust Games CollaborativeLearning TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication CommunicationStudies SoftwareArchitecture

  3. ... Conversational Agent, that is, an agent that can participate in fully natural dialog ... - James Allen Conversational Agents exploit natural-language technologies to engage users in text-based information-seeking and task-oriented dialogs for a broad range of applications. - James Lester Conversational agent is a system that exchanges information between user and agent using natural language dialogue - Sofus Macskassy Conversational agents are communication technologies that use natural language and computational linguistic techniques to engage users in human-like, Web-based dialogs. - Wikipedia Conversational Agents (CAs) • General Definition Conversational Agents are automated agentsthat extend conversation as a medium ofinteraction with machines. • Many Applications > Background > Conversational Agents

  4. Some Conversational Agents • Many studies have shown effectiveness of CAs • Information Access > Raux et. al., 2005 • Intelligent Tutoring > Kumar et. al. 2006/2007a • Therapy > Bickmore et. al., 2005 > Background > Conversational Agents

  5. Bridges Multi-PartyInteraction Social /Trust Games CollaborativeLearning TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication CommunicationStudies SoftwareArchitecture

  6. Multi-Party Interactive Situations (MPIS) • Multi-Party Interactive Situations • Meetings, Dinner, Games, Classrooms • Groups more effective than Individuals at Intellective Tasks Review in McGrath, 1984 • Increasing Computer Mediated Multi-Party Interaction • Instant Messaging / Chat • Video Conferencing • Collaborative Learning • Beneficial pedagogical approach • Multi-player Online Games • Collaborative Work • Online Auctions • Social Networking > Background > Multi-Party Interactive Situations

  7. Bridges Multi-PartyInteraction CollaborativeLearning Social /Trust Games TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication CommunicationStudies SoftwareArchitecture

  8. CAs in Multi-Party Interactive Situations • One Agent + Two or More Users • E.g.: Tutor supporting Collaborative Learning Moderator in Group Interaction • Two or More Agents + One User • E.g. TeamTalk Harris & Rudnicky, 2007 • Two or More Agents + Two or More Users • E.g.: Multi-player Games (with multiple NPCs) > Background > Conversational Agents in Multi-Party Interactive Situations

  9. CAs in Multi-Party Interactive Situations • One Agent + Two or More Users • E.g.: Tutor supporting Collaborative Learning Moderator in Group Interaction Existing Work > Background > Conversational Agents in Multi-Party Interactive Situations

  10. CAs in MPIS: Two Challenges • Building / Implementing Agents for Multi-Party Interactive Situations • Technical Challenge • Engineering / Development Issues • Designing Suitable / Useful Agent Behaviorfor such Situations • Scientific Challenge • Design Principles / Guidelines > Problem Statement

  11. Technical Challenge: Building CAs • Existing approaches • Architectures • Backboard, Pipeline, Hub, Multi-Expert, … • Representations • State, Frame, Schema, Agenda, Plan, … • Multi-Expert Architectures • Jaspis Turunen & Hakulinen, 2003 • RIME Nakano et. al., 2008 • Event-driven Dialog Management • Olympus 2.0 > Interaction ManagerRaux & Eskenazi, 2007 > Problem Statement > Building CAs

  12. Technical Challenge: Building CAs • Shortcomings of Existing approaches • Poor Representational Capability • Inflexibility to handle Complex Interaction Dynamics > Problem Statement > Building CAs

  13. Technical Challenge: Building CAs • Poor Representational Capability • E.g. Scheduling policy for simultaneous responses from multiple users • Very High Level Languages • Restricted to small set of operators and control structures • Inability to freely combine representations in existing architectures > Problem Statement > Building CAs

  14. Even Participation Assumption Single User Interaction Multi-Party Interaction System Technical Challenge: Building CAs • Inflexibility to handle Complex Interaction Dynamics • E.g. • Ensuring all users get an equal opportunity to contribute • Two Assumptions that Fail > Problem Statement > Building CAs

  15. Known Addressee Assumption • Schisming Student1 OK, lets start Tutor What would happen to the power output of a Rankine Cycle at a higher operating temperature? Student2 hmmm … Can you answer that? Student1 I think it will increase. Tutor The correct answer is that at a higher operating temperature, more heat is added to the cycle and hence the power output increases too. What about the heat rejected by the cycle though? ? ? Tutor You are right S1. It increases too. Lets move on to the next topic. Technical Challenge: Building CAs • Inflexibility to handle Complex Interaction Dynamics • E.g. • Ensuring all users get an equal opportunity to contribute • Two Assumptions that Fail > Problem Statement > Building CAs

  16. Technical Challenge: Contributions • Basilica A novel software architecture (and development tools) for building CAs in MPIS that provides • Rich Representational Capability • Using High-Level Language • Flexibility to address Complex Interaction Dynamics • Does not make Even Participation and Known Addressee assumptions • An array of CAs built using this architecture that provide • Re-useable components > Problem Statement > Building CAs > Contributions

  17. Second Life Middleware OutputCoordinator SLActor SLListener TutoringActor TurnTakingCoordinator MessageFilter GreetingActor TutoringManager TouchFilter Basilica: Software Architecture • Event-driven Architecture • Agent Representation Kumar & Rosé, 2009 • Network of Behavioral Components • Components are programmable using High-level Languages • Analogy: UI Toolkits • E.g.: Java Swing > Building CAs > Basilica

  18. Basilica: Software Architecture • Core Classes: Provide implementations for • Agent & Component Management • Event Propagation • Observer Interfaces • Agent Factory: Runtime Agent Assembly (from XML) • Utilities & Development Tools • Re-Usable Components > Building CAs > Basilica

  19. Basilica: Four Agents • Four Agents built using Basilica • CycleTalk Tutor Agent • PsychChallenge Peer Agent • WrenchTalk Tutor Agent • Emergency Response (911) Interpreter Agent • Four different Multi-Party Interactive Situations > Building CAs > Basilica > Agents Developed

  20. CycleTalk Tutor Agent > Building CAs > Basilica > Agents Developed

  21. CycleTalk Tutor Agent • Implements Turn-taking rules suitable for Tutoring situation • Wait for responses from students • Prioritize between correct/incorrect/irrelevant responses ConcertChat Server OutputCoordinator ConcertChatActor ConcertChatListener PromptingActor AttentionGrabbingActor AttentionGrabbingFilter TutoringActor TutoringManager RequestDetector TurnTakingCoordinator MessageFilter HintingActor HintingManager > Building CAs > Basilica > Agents Developed

  22. PsychChallenge Peer Agent On Learning Portal of major Publishing Company > Building CAs > Basilica > Agents Developed

  23. HTTP PsychChallenge Web Interface (API) Generic HTTP Middleware (Servlet) M.WareActor OutputCoordinator M.WareListener GreetingActor StatusFilter RoleActor RoleFilter HintingActor ScoreMemory GuessingActor WordMemory PsychChallenge Peer Agent • Agent has the same role as users > Building CAs > Basilica > Agents Developed

  24. WrenchTalk Tutor Agent > Building CAs > Basilica > Agents Developed

  25. WrenchTalk Tutor Agent ConcertChat Server ConcertChatActor ConcertChatListener MessageFilter PresenceFilter DiscourseMemory AnnotationFilter OutputCoordinator SocialController ActivityDetector ProgressDetector PlanExecutor RequestDetector T.TakingCoordinator IntroductionsManager PromptingManager TutoringManager TutoringActor IntroductionsActor PromptingActor > Building CAs > Basilica > Agents Developed

  26. NineOneOne Interpreter Agent V I D E O > Building CAs > Basilica > Agents Developed

  27. XMPP Connector XMPPActor XMPPListener PresenceFilter Caller Proxy Live Audio ASR TTS MessageFilter TellCallerActor CallerTurnMemory DistressedCaller NLUFilter MetaActor RequestTypeFilter IQAActor MetaActManager Parser XMPP Server (Communication Backbone) ActClassifier IQAActManager SlotConfirmationActor SlotConfirmationManager GenericsDetector RequestDetector InformDispatcherActor Emergency Dispatcher ReportDetector SlotValueDetector NineOneOne Interpreter Agent > Building CAs > Basilica > Agents Developed

  28. CAs in MPIS: Two Challenges • Building / Implementing Agents for Multi-Party Interactive Situations • Technical Challenge • Engineering / Development Issues • Designing Suitable / Useful Agent Behaviorfor such Situations • Scientific Challenge • Design Principles / Guidelines > Problem Statement

  29. Scientific Challenge: Agent Behavior • Unlike individual learners,Teams of students ignore / abuse automated tutorsKumar et. al. 2007a • Agents lack Social Communication Skills • Need to be better communicators > Problem Statement > Agent Behavior

  30. Interaction & PerformancePhase MovementConflict Resolution Bridges Multi-PartyInteraction CollaborativeLearning Social /Trust Games TutorialDialog CSCW/CMC DialogSystems My Thesis ConversationalAgents Small GroupCommunication SoftwareArchitecture CommunicationStudies

  31. Scientific Challenge: Agent Behavior • Small Group Communication • Two Fundamental Processes • Bales, 1950 (Problem Solving Groups) • Instrumental (task-related) vs.Expressive (social-emotional) • Need for an Equilibrium • Developed Interaction Process Analysis (IPA) • Bion, 1961 & Thelen, 1956 (Therapy Groups) +ve Expressive Instrumental -ve > Problem Statement > Agent Behavior > Small Group Communication

  32. Scientific Challenge: Agent Behavior • CAs must perform both Task-related as well as Social Interaction • Task-related Interaction strategies • Our earlier work Chaudhuri et. al., 2008, 2009 • Social Behavior > Problem Statement > Agent Behavior

  33. Scientific Challenge: Social Behavior Related Work: • Verbal Social Behavior by CAs • Affective Computing • Non-Verbal Social Behavior by ECAs Multi-Party > Problem Statement > Agent Behavior > Related Work

  34. Scientific Challenge: Contributions • Development of Social Interaction Strategies motivated from research in Small Group Communication • Investigation of effectiveness of Social Interaction Strategies in MPIS • Experiments on amount, timing and type of behavior • Studied in multiple task domains • Collaborative Learning & Trust Games • Methodology & Materials to further the development of CAs as Good Communicators • Towards incorporating research from Small Group Communication in CA design > Problem Statement > Agent Behavior > Contributions

  35. Social Interaction Strategies • Eleven Social Interaction Strategies developed • Based on Three Positive Social-Emotional Interaction Categories > Social Behavior > Design

  36. ConcertChat Server ConcertChatActor ConcertChatListener MessageFilter PresenceFilter DiscourseMemory AnnotationFilter OutputCoordinator SocialController ActivityDetector ProgressDetector PlanExecutor RequestDetector T.TakingCoordinator IntroductionsManager PromptingManager TutoringManager TutoringActor IntroductionsActor PromptingActor Implementation: WrenchTalk Tutor Agent • Two primary controllers • PlanExecutor > Executes Task-related steps • SocialController > Triggers Social Behavior • Controllers regulate each other > By Blocking > Social Behavior > Implementation

  37. 1d. 2b. 2b. Implementation: Social Controller • Social Behavior Triggering • Hand Crafted Rules • Four Features • Last executed plan step • Annotations of student turns • Dictionary Lookup • Activity Levels • Groups & Individual • Strategy: 1e. (Encourage) • Social Ratio • Ratio of Social Turnsto Task-related turns • Threshold: 20% > Social Behavior > Implementation

  38. > Social Behavior > Experiments

  39. Research Questions: Experiments • Effectiveness of Social Behavior Vs. • Gold Standard (Human performance) • Baseline (No Social Behavior) • Right Amount of Social Behavior • Contribution of individual behaviors • Effect of the Triggering Model • Generalizability to other Interactive Situations Experiment 1 Experiment 2 Experiment 3 ProposedTask 1 & 2 ProposedTask 3 & 4 > Social Behavior > Experiments

  40. Experiment 1-3: Interactive Situations • Collaborative Design Labs • Mechanical Engineering Courses • Freshmen: Wrench Design • Teams of 3-4 students • Underlying Concepts • Force, Moment, Stress, Strength, … • Sophomore: Power plant design • Teams of 2 students • Underlying Concepts • Relationships between 4 System Properties & 5 Response Variables • Tasks structured using Worksheets > Social Behavior > Experiments > Situations

  41. Experiment 1-3: Methodology • Controlled Experiments • Between subjects • Conducted over multiple sessions • 35-40 minutes per session • Different students in each session • Students randomly assigned to teams on the spot • Team mates not seated next to each other • Communicate using ConcertChat • Teams randomly assigned to conditions • Nearly even distribution of conditions in each session • Incentive • Class Credit • Gift Certificates for best team designs > Social Behavior > Experiments > Methodology

  42. Experiment 1-3: Labs > Social Behavior > Experiments > Metrics

  43. Experiment 1-3: Metrics • Performance: Learning Outcomes • Pre & Post Tests • Multiple choice questions (MCQs) • Short Essay Questions (SEQs) • Perception: Survey Burke, 1967 • 7-point Likert-scale (1-Strongly disagree, 7-Strongly Agree) • 9 items • Ratings about Tutor (Agent): Likeable, Friendly, … • Satisfaction Ratings: Task & Group Discussion • Legitimacy > Social Behavior > Experiments > Metrics

  44. Research Questions: Experiments • Effectiveness of Social Behavior Vs. • Gold Standard (Human performance) • Baseline (No Social Behavior) • Right Amount of Social Behavior • Contribution of individual behaviors • Effect of the Triggering Model • Generalizability to other Interactive Situations Experiment 1 > Social Behavior > Experiments

  45. Experiment 1: Evaluating Effectiveness • Experimental Design • Three Conditions • Benefit of the Collaborative Design Activity • No significant difference between pre-tests for the three conditions • Significant improvement from pre-test for post-test for all groups • F(1,190) = 16.67, p <0.001, Effect size = 0.51σ > Social Behavior > Experiment 1

  46. Experiment 1: Results > Performance Learning Outcomes • ANCOVA post-test = Fn(1, pre-test, condition, day) • Significant effect of Condition F(2, 93) = 10.56, p < 0.001 • Post-hoc Analysis • Task vs. Social { p < 0.01, 0.71σ } • Task vs. Human { p < 0.001, 0.93σ } • Social vs. Human { p = 0.509 } • Tests: 8 MCQs & 3 SEQs Analysis done on total score > Social Behavior > Experiment 1 > Results

  47. Experiment 1: Results > Perception • Human & Social conditions better than Task conditions • Human condition significantly better on (Q1 - Q5, Q8) • Social condition significantly beter on Q2, marginal on Q4, Q5 > Social Behavior > Experiment 1 > Results

  48. Experiment 1: Summary • Significant benefits of employing Social Interaction Strategies • Implementation of Social Tutors not as good as Human tutors • Right Amount of Social Behavior • Significantly more Social Behavior in Human condition Average (Human) = 22.17 Average (Social) = 16.83 • Human Social Behavior Triggering Experiment 2 ProposedTask 1 & 2 > Social Behavior > Experiment 1

  49. Experiment 2: Amount of Behavior • Manipulated by changing Social Ratio • Experimental Design • Three Conditions • Benefit of the Collaborative Design Activity • No significant difference between conditions on pre-tests • Significant improvements from pre- to post-test • Effect size = 0.79σ (Total), p < 0.001 > Social Behavior > Experiment 2

  50. Experiment 2: Results Performance > Learning Outcomes • Tests: 22 MCQs & 6 SEQs • ANCOVA post-test = Fn(1, pre-test, condition, session) • Significant effect of Condition on MCQs F(2, 97) = 3.48, p < 0.05 • Post-hoc Analysis (for MCQs) • None vs. Low { p < 0.07, 0.69σ } • Low vs. High { p < 0.07, 0.55σ } • None vs. High { not significant } Perception > Survey • No Significant Differences between conditions Use moderate amount of Social Behavior > Social Behavior > Experiment 2 > Results

More Related