Communication Working Group Microsoft Research & ALS Society of

Aiming Computing Technology at Enhancing the Quality of Life of People with ALS Some Sketches on Directions in Minimal Signaling Communication Communication Working Group Microsoft Research & ALS Society of British Columbia

Directions • Tools and method for planning and creating long-term corpora and strategies for communication • Creating effective input strategies and systems for different phases of the progression of ALS • Exploration of novel signaling strategies, such as eye gaze, as an effective input modality • Coupling contextual information as source of evidence in reasoning about intentions • Potential for integrated robotic systems for motion • Multiple directions

Core Challeges & Opportunities with Gaze-Centric Systems • Move beyond potentially frustrating point and dwell approaches • Toward new UI designs, metaphors & control methodologies that can ease user effort and enhance accuracy of selection, control. • Adaptive techniques, use of context can ease user effort and error. • Promise of creating multiple rewarding and valuable applications, for communication, creative expression, information access, enjoyable experiences, control of real-world sensors and effectors.

Sample Applications

Example • Gaze-centric interaction with (previously authored) databases of images and audiovisual snippets for enhanced communication

Social discourse library of videos and stills for sharing emotions, moods, gestures. Memories library from video and image photolibrary

Example • Gaze-centric control of confirmation, negation, selection of alphabetic controls for writing, editing.

Q W E R T Y U I O P A S D F G H J K L Z X C V B N M . ? [ I want to … No way. Big Time! [ask] [tell] [have] Predictive language model with correction No Yes

Example • Gaze-centric control of integrated, supportive robotic systems • Head motion • Head gestures (nods, yes; shakes no) • Arms (and hand?) motion …etc.

Turn head left or right (e.g., in increments)

Conversation Confirmation Nod “yes”; shake head “no” (Beyond confirmation: nod as a natural cue for understanding in stream with conversation)

Toward more general application of “integrated robotics,” e.g., arms controls, etc., for atomic and patterned moves (e.g., wave hello).

Example • Gaze-centric control of navigation through cached real-world, or 3D virtual-reality animation sequences, potentially with other appropriate sensory stimulation.

Example • Gaze-centric communication and linking for single and multiplayer games, including chess, card playing, virtual worlds (Second Life, etc.)

Example • Gaze-centric control of telepresence applications for real-time viewing from different viewpoints.

Example • Gaze-centric control of painting, CAD, sculpting machinery, machine lathes, other artistry and crafts.

Longer-Term: Addressing Totally Locked in State (TLS) • Challenge of use of EEG for preference assessment, acknowledgment, confirmation, selection • E.g., Rotating selection rings, with “other” option; navigation among hierarchy of rings, use of simple input for halting, selecting, confirmation.

Multiple Directions and Possibilities • Comments? horvitz@microsoft.com http://research.microsoft.com/~horvitz

Communication Working Group Microsoft Research & ALS Society of