Architectures and Standards for IVAs at the Social Cognitive Systems Group

1. Architectures and Standards for IVAs at the Social Cognitive Systems Group H. van Welbergen, K. Bergmann, H. Buschmeier, S. Kahl, I. de Kok, A. Sadeghipour, R. Yaghoubzadeh, S. Kopp

2. Main research themes • How can cognitive systems be designed as intelligent, socially adept interaction partners that allow a fluent and coordinated interaction with humans • Developing methods to model the behavioral, preceptual motor and cognitive mechanisms of embodied human-like communication and cooperation • Application and evaluation in IVAs • Scenarios: • Embedding IVAs in mouse-keyboard interaction • Virtual coaches • Virtual assistents for the elderly and cognitively impaired • Cognitive models that investigate semantic speech/gesture coordination • Computational models of dialog coordination based on linguistic feedback

3. Current Architecture and Standards • Our components: • A multimodal memory component realized as a spreading activation model of semantic coordination for speech-gesture production (Bergmann et al., 2013) • Incremental NLG based on SPUD (Buschmeier et al., 2012) • A Behavior Planner for iconic gestures (Bergmann and Kopp, 2009) • A BML 1.0 Realizer for incremental and adaptive behavior (van Welbergen et al. 2014) • An IS-based incremental dialog manager, capable of handling uncertain input (yet unpublished) • Several external components • Often multiple alternatives available, allowing trade-offs between recognition/synthesis quality, reactivity and control

4. Current Architecture and Standards • Combining components • We follow the SAIBA reference architecture • BML 1.0 for realization, extensions for incrementality and adaptivity • No standardized communication (via FML) between the Intent and Behavior Planner • Components • Run on various programming languages, OSs, computers • Allow incremental processing • IPAACA • Implements the Incremental Unit architecture (Schlangen and Skantze, 2011) • Embeds it in a message oriented middleware (RSB; Wienke and Wrede 2011)

5. Future Architectures and Standards: LONG TERM • Base our architecture on universal (less problem specific) and cognitively motivated principles • Ongoing work: • Fully incremental production an recognition and production, e.g. in attentive speaking (Buschmeier et al 2012) • Representation and decision making under uncertainty • Using uncertainty as valuable information rather than as a mere nuisance • Priming and imitation learning of gesture (Sadeghipour and Kopp, 2011)

6. Future Architectures and Standards: short term • SAIBA provides us: • A common terminology for large scale IVA components • A standardized interface for behavior realization • We would like to enhance it to a full reference architecture for IVAs • Our requirements • Handle both input processing and output generation • Coordination between these at multiple levels • Incremental input and output processing • Representing memory • Representing uncertainty

7. Future Architectures and Standards Asap(Kopp et al., 2014)

8. Suggestions for discussion: a new reference architecture for IVAs • Our requirements • Handle both input processing and output generation • Coordination between these at multiple levels • Incremental input and output processing • Representing memory • Representing uncertainty • Requirements of others • Representing and interacting with ‘the world’ • Generalization to multi-agent/multi-human

9. Suggestions for discussion: a new reference architecture for IVAs • Desired outcome: • Prioritized list of requirements, plan to incrementally embed them • Definition of shared terminology • Architecture sketch, identification of shared interfaces • Agenda for shared interfaces between modules • Workshops/plans of progress for BML, FML, PML, …

10. Suggestions for discussion: sharing and combining smaller components • Many interesting ‘small’ IVA components are presented every year • In isolated projects/experiments • How can these be embedded in the larger effort of developing a full IVA? • We are especially interested in doing this for the Behavior Planner • The challenge is not only in implementing individual parts that are potentially useful but also in combining the components

11. Suggestions for discussion: sharing and combining smaller components • Desired outcome: • (Start of) design pattern catalogue for the design of (parts of) a behavior planner • E.g. MAS, blackboard, MOM, IU-architecture, subsumptionarchitecture • Ideas to share smaller components • List of good candidate components for sharing • Inspired by best practices from related fields (e.g. ROS: Quigley et al., 2009) • Can we develop standardized components using challenges for each of them?

12. Suggestions for discussion: challenges • Goal: • Foster the development and comparison of reusable IVA components • Different from the goals of IVA’s GALA • Provide a forum to present the state-of-the art in technology of virtual humans and innovative applications • Aimed to stimulate (under)graduate student participation • Disseminate demos in an ever evolving gallery for study and reuse • In practice the “dissemination” was limited to a video

13. Suggestions for discussion: challenges • Desired outcome: • GALA post-mortem • Best practices of successful challenges in related fields • RoboCup@Home, Blizzard TTS, Give NLG, INTERSPEECH, … • Requirements/guidelines for a challenge • Plan to embed challenge in a workshop/conference (IVA?) • Example design (sketch) of a first challenge

14. Thanks for your attention • Questions?

15. Suggestions for discussion: challenges • Challenges in related fields: • RoboCup@Home, Blizzard TTS, Give NLG, INTERSPEECH, … • Some interesting ideas from these: • Shared set of ‘resources’, fosters comparable system building • E.g. recorded voices to build off (Blizzard), shared environment to navigate through (Give, RoboCup@Home), shared corpus (INTERSPEECH) • Given by the organizers of the challenge • Predefined set of automatically obtained evaluation criteria (GIVE, INTERSPEECH) • Baselines (INTERSPEECH) • ‘Sharing’ subjects (Blizzard, RoboCup@Home) • Both the challenge and the submissions are full-fledged, peer-reviewed papers (INTERSPEECH challenges, Blizzard challenges) • Mandatory (but delayed) publishing of source code and technical documentation (RoboCup@Home)

16. Suggestions for discussion: sharing and combining smaller components • Inspiration from related fields: ROS (Quigley et al., 2009) • Provides over 3000 robotics components • Provides the ‘plumbing’ of robotic systems • Connecting components: • Publish subscribe, rpc, actions, distributed parameter setting, IDL’s for messages, starting components, central time, … • Debugging connections: visualization, recording/playback • Building and distributing components: • Tools for building and distributing ROS-packages • We have reinvented the wheel on several of these… • Some adhering to conventions required