Denis Gouin Valérie Lavigne Alexandre Bergeron-Guyard Innovative Interfaces and Interactions Group

1. Human-Computer Interaction with an Intelligence Virtual Analyst Denis Gouin Valérie Lavigne Alexandre Bergeron-Guyard Innovative Interfaces and Interactions Group Intelligence and Information Section DRDC Valcartier KSCO 2012 February 2012

2. iVAC initiative • Intelligent Software Assistant (ISA) identified by MIT’s Technology Review as one of 2009’s most promising emerging technologies • Conversational, computer-generated characters capable of providing guidance to a user in the conduct of his tasks • Intelligence Virtual Analyst Capability (iVAC) R&D initiative at Defence R&D Canada • 3-year initiative started in 2011 • Knowledge System with important HCI component

3. Virtual Assistant / Virtual Advisor DARPA’s CALO Cognitive Assistant that Learns and Organizes IBM's Watson supercomputer SIRI Apps on IPhone 4s

4. DARPA’s Personal Assistant that Learns (PAL) Video Clip presentation http://www.youtube.com/watch?v=BF-KNFlOocQ&feature=youtu.be

5. Cognitive Assistant that Learns and Organizes (CALO)

6. HCI Requirements - General • HCI dialogue between iVAC and the analyst(s) • How to input knowledge into the system • How to raise questions • How to task the IVA • How does the IVA ask for precisions on the questions and the way of presenting information • Optimization of the presentation of the results • One and/or multiple virtual analysts can serve multiple users • Virtual analysts can interact with each other within and across meeting spaces

7. Enabling HCI Technologies • Smart Room Environments • Multimodal Interaction • Information Presentation and Adaptive Interfaces • Augmented Cognition • ISA Representation / Avatars • Storytelling

8. Smart Room Environments • Instrumented environments (e.g. cameras, microphones, biometry) • Ubiquitous computing. The user can access the system from various locations • Track users location • Track users activities US Navy Command Center of the Future (CCoF) DSTO LiveSpaces

9. Multimodal Interaction • Allow the user(s) to employ multimodal interaction (e.g. voice, pointing, gesture, eye/gaze, neural/brain interfaces, emotion detection) • Resolve ambiguities • These are my priorities • We should move that milestone back 6 months Xbox Kinect AFRL Interactive Data Wall Emotion Detection, Valenti et al. 2007

10. Information Presentation • In support of various tasks • Present tools and information • Answer questions • Ask for clarification • Remind the user of some tasks or procedures • Propose alternative possibilities • Give a briefing

11. Information Presentation • Means • Voice output and/or information display • Highlight information elements • Display information in a new window, organized as the user wants to see it • Gather a set of documents • Filter information based on user requests • Capture information about a meeting (automatic transcription and semantic analysis of multiparty meetings )

12. Adaptive Interfaces • Customized to the role and current tasks of the user • Present tools and information based on the user preferences • Suggest sequences of events based on learned tasks and preferences • Understand the context / tasks • Observe what the user is doing and his mental state • Do not disturb • The user is recognized using biometry • The interface adapts to the distance of the user to the (large) display (text fonts and granularity of information)

13. Augmented Cognition • Track the sensory and cognitive overload of the users • Use of neural/brain interfaces, emotion detection • Employ computational strategies to restore operational effectiveness • Intelligent interruption to improve limited working memory • Attention management to improve focus during complex tasks • Cued memory retrieval to improve situational awareness and context recovery • Modality switching (i.e., audio, visual) to increase information throughput • When not to disturb DARPA (2011a), DARPA Defense Science Office, Improving Warfighter Information Intake Under Stress (AugCog), http://www.darpa.mil/dso/archives/warfighter/index.htm

14. ISA Representation / Avatars • Various representations • Speech output (voice) • A simple icon • A two- or three-dimensional representation of a character (avatar) PAL

15. ISA Representation / Avatars • Adaptation of the avatar to the task and socio-cultural context • Facial and voice features of the avatar (serious / smiling, tone of the voice ) should communicate emotions, danger, risk, uncertainty • Consider socio-cultural factors (Gender / age / ethnicity / profession) • Use different avatars to support different ISA tasks (weather prediction or course of action) • Embedding avatars in a video conferencing • Be careful of the Spectre of the Uncanny Valley • Need for experiences with avatar representations DSTO’s FOCAL www.postgenomic.com/faces

16. Storytelling • Guide audience through a preconstructed narrative • Use of tangible graphics and videos (TV news like) • Richness of the storytelling approach • Convey situation awareness about complex events Logo – VisWeek -Telling Stories with Data

17. ISA for Coalition Operations • Improved interoperability between coalition forces in terms of disseminating information • Translate information between languages • Share differences in tactics, techniques and procedures (TTPs) • Synchronizing coalition activities • Improve cognitive assistance by providing shared coalition awareness and task support • Enable coalition organizations to learn by managing a knowledge base • Provide better and faster decision making through access to comprehensive knowledge developed through time

18. Conclusion • ISA, a knowledge system component • Solve the human cognitive overload • Conducting a wide variety of tasks • Search and organize information • Learn procedures and preferences • Manage schedules • Track people • HCI is Key • Multimodal interaction • Avatar is not mandatory • Assign tasks • Summarize documents • Mediate interactions • Guide and remind the user