A Speaking Assistant for Personal Information Management

1. A Speaking Assistant forPersonal Information Management Second Year Thesis Proposal Presentation Nuno Jorge Gonçalves de Magalhães Ribeiro Supervisor: Ian Benest Assessor: Patrick Olivier Department of Computer Science University of York July 2000

2. Motivation (1/2) Speech employed in multimedia interfaces is useful: • when information is visually hidden (e.g. information lower in hierarchy) Interactive Guide (interactive multimedia presentations) • when information is visually disruptive (e.g. description of a diagram) Reportage (summarising reports including multimedia presentations) • when various kinds of reminding and alerting information must be reported Monitor reports: notifications (unpredictable events) reminders (alarms with semantics) Thesis Proposal

3. Motivation (2/2) Open issues to investigate: • When does speech enhance the user interface and when is it detrimental to the user interface? • Monitor reports interrupt: • the user’s activity • the audio channel (reportages, interactive guide) How should then these interruptions be handled so as to minimise disruption on the user? Thesis Proposal

4. Social Interfaces (Nass and Reeves) People react socially to mediated interfaces in much the same way as they react in face-to-face situations If interfaces exhibit supportive modalities that cue social responses Then people perceive the interaction as more natural Humanised Interface Thesis Proposal

5. Characteristics of a humanised interface • Interruptions (urgency, priority) • Politeness (appropriate level) • Discourse Variation (context = user activity) • Message Aggregation (summary, less interruptions) • User Presence Late Messaging Thesis Proposal

6. Thesis Proposal: Hypothesis The incorporation of speech, conveyed using human discourse characteristics, can improve a computerised work environment in which the computer is perceived to be a work companion. Thesis Proposal

7. Thesis Proposal: Hypothesis (1/2) 1] Perception of a work companion use human discourse characteristics • discourse variation • politeness • message aggregation The use of human discourse characteristics such as discourse variation, politeness and message aggregation promotes the perception of a work companion. Thesis Proposal

8. Thesis Proposal: Hypothesis (2/2) 2] Work Environment characteristics proper management of interruptions • interrupt as a human would do • don’t avoid interrupting, but... • minimise disruption • interrupt the audio channel A proper management of interruptions occurring in the computer-based work environment promotes the perception of a work companion. Thesis Proposal

9. Towards a demonstrator • Develop a Speaking Assistant architecture to allow for: • a number of specialised agents (e-mail, diary, printer) • a speaking agent • Speaking agent behaviour: • creates spoken monitor reports (reminders, notifications) corresponding to messages received • creates reportages (structured multimedia reports) • interrupts: grabs user attention finds an appropriate place to stop a reportage presents a monitor report restarts the reportage at an appropriate place Thesis Proposal

10. Towards a Personal Assistant in a Multimedia Environment Personal Assistant (Specialised Agents) Speaking Agent infrastructure architecture e-mail agent user activity monitor (context) Reportage Interactive guide ... diary agent Dispatcher Aggregator Vocaliser MM Engine User printer agent user presence monitor tannoy agent Prioritised Messages Notification Reminder Reportage Sentences Spoken messages Scheduler Priority heuristics Template-based sentence generation Intonation Prosody generation Attention-grabbing Audio channel interruption Thesis Proposal

11. Contributions (1/2) How to interrupt the user (activity + audio channel) • grab the user’s attention to notify / remind • fade out, report, resume at appropriate locations must be handled at the last stage of the process When to interrupt • notification mechanism with interruption levels that depend on urgency, priority and user activity must be provided by specialised agents Thesis Proposal

12. Contributions (2/2) What is an appropriate architecture to support a multi-agent personal assistant? • a number of specialised agents • handle specific parts of the user work environment • send notifications/reminders to be spoken by the speaking agent • sense the user presence • a speaking agent • speaks for a number of specialised agents • dispatches, aggregates, vocalises and interrupts • a suitable inter-agent protocol • what information must specialised agents provide? Does this architecture promote the perception that a system is humanised? Thesis Proposal

13. Evaluation Assessment of the importance of the incorporation of human discourse aspects at the interface • Is the system useful? Will people continue to use it? • Are the generated interruptions appropriate? • Is the way used to interrupt appropriate? • Does it appear to be a work companion? Thesis Proposal

14. Work Plan Thesis Proposal

15. Thesis Outline (1/2) • Part I: Introduction • Chapter 1: Speaking Assistant for PIM • Part II: The Problem and its context • Chapter 2: Multimedia User Interfaces • Chapter 3: Problem Space • Chapter 4: Related Work • Chapter 5: Project Outline Thesis Proposal

16. Thesis Outline (2/2) • Part III: The Prototype • Chapter 6: Specification and design • Chapter 7: Implementation • Chapter 8: Empirical Study • Part IV: Final Remarks • Chapter 9: Conclusions • Chapter 10: Summary and future work Thesis Proposal