Sensing Techniques for Mobile Interaction

1. Ken Hinckley Jeff Pierce Mike Sinclair Eric Horvitz Attentional User Interfaces Project Microsoft Research Sensing Techniques for Mobile Interaction

2. Mobile Devices vs. Mr. Cleo (An Informal Comparison) • How can we make Smart Computers? Mr. Cleo • Aware of sounds, objects • Knows if I walk in the room • Selfish & Inconsiderate • My Mobile Device • Unaware of environment • Oblivious to my presence • Selfish & Inconsiderate

3. Sensitive Interfaces(a.k.a. “Not-So-Stupid Computing”) • Computers are very fast idiots. They are oblivious to the external world. • What is a “smart computer” anyway??? • Can we build one? Do you really want one? • Or one that is aware, respectful, … Not So Stupid? • We’d be better off if computers were as dumb as my cat • Can some very dumb sensors, with straightforward software, help address this problem? • Design hardware/software interfaces that are sensitive to the user and the surrounding physical environment.

4. Sensing for User Interaction • Sense more than just explicit commands • Simplify the interface using background informationthat is already there • Point+Shoot Cameras • One Button • Many sensors

5. Can “Background Sensing” Enhance Mobile UI? • Real world has high cognitive / attentional demand • Even “clicking a button” can be hard! • Software ignorant of the changing physical context • Device should adapt to current task / situation • Naturally occurring “gestures” of use are missed • Pick up, put down, look at, walk around with… • Explosion of cheap, informative sensors • On device or available via wireless network

6. Mobile Sensor Prototype (Casio E105) • Tilt Sensor • Proximity Range Sensor • Touch Sensor • Demos implemented as Windows CE applications • Sensor I/O via PIC micro. • VIDEO

7. Tilt Sensor • Analog Devices 2-axis linear accelerometer • Tilt relative to gravity • But, other accelerations also affect signal • Limitations: • Cannot sense rotation around vertical axis • Cannot tell up from down

8. Activity Detection: Tilt Sensor Example Data Looking at display Walking to elevator Walking to meeting PocketPC held at side

9. Voice Memo Detector • Gesture allows general-purpose PIM to have special-purpose context of use • Sensor Fusion: • Must be holding device • Tilt “like a phone” • Hold close to face • Audio is crucial to the interaction • Relaxing stops recording

10. Normal approach Sensor approach • Pick up device • Find the ¼” dia. button • Position hand to press button • Press & maintain tension • Listen for beep • Record your message • Release when done • Double-beep confirms • Pick up device (to face) • Listen for the beep • Record your message • Relax device when done • Double-beep confirms recording was made Voice Memo:Workflow Analysis • “I have to think about finding the button, pushing .it, holding it” • “It was just listen for the beep”

11. Sensed Memo Recording: Usability Issues • Sensed gesture not easily discoverable • “disorienting to put up to my face to talk” - 1 Ss • False positives can occur: e.g. putting into sweater pocket • But, button also has false positives… • “talk into it like a cell phone” is enough instruction to use • “Quite a bit easier, I can focus” • “Would use it more if it worked that way” • 6/7 Ss preferred sensed gesture to button (4.3)

12. Tracking Experiment • Not faster, but less demand on visual attention. • User tracks “fly” on monitor, using mouse • Records “Testing 123” • (S, M) > C (p<.001) • Sensed gesture may require less visual attention (p < 0.1)

13. Portrait / Landscape Display Mode Detection • “Snow Globe” • Input controls rotated to match screen • Easy (5.0/5.0). • 6 Ss prefer tilting to menu; 1 Ss “I think it would drive me nuts” • Sharing w/others • Doesn’t interrupt

14. Portrait / Landscape Detection: Implementation Tilt vs. sensed orientation: • 5 display modes • 2 Portrait • 2 Landscape • Flat • Dead bands, 0.5s: keep screen stable • Put-down problem: FIFO queue, look for stable orient.

15. Tilt for Scrolling • Touch screen bezel to clutch • Easy to activate / maintain state • Too easy to hit by accident (landscape mode) • Sets “resting” orientation • Hides on-screen UI (menu, taskbar) during scrolling • Several transfer functions possible • Rate ctrl: v = K * sgn(dA) max(||dA|| - dAmin, 0) • Single axis: only move along 1 primary axis at a time • Dual axis: full panning in any direction • Mixed axis: panning, but with affinity for primary axes

16. Tilt: Scrolling &Then Some • Contrast compensation • Equalize “apparent contrast” • Scrolling + Portrait/Landscape • Don’t change P/L modes while scrolling! • Don’t change when stop scrolling, either! • Waits to see different orientation • Not quite right; should switch after longer dwell (2-3 s?) • User Testing: 5 Ss, compared to built-in direction pad • “Good way to scroll the screen” – Agree (4.8 / 5.0) • “I’d rather use the direction pad” – Disagree (1.8 / 5.0) • One-handed operation, natural, simplifies the movement

17. Power Management • Automatic power-on • Holding device in hand • AND looking at display (flat in one axis, tilted fwd ~20o in the other) for 0.5s • Can’t power up in purse / pocket • Won’t power up if you just touch it to push away • Won’t power up if you just grab & hold at side (usually ) • There is no auto-power-OFF feature, by design • Best case: turns off & user doesn’t even notice feature • Touch, tilt & proximity sensors prevent power-off or screen dimming while using device

18. Other Fun Stuff • Games • Proximity • For zooming – fun demo, but how to select? • Proximal UI: Creepy • “Hand of frustration” • Shaking • To switch applications (top app  bottom) • Turn upside down & shake to erase :-) • Impl. not robust enough to turn on all the time

19. Conclusions • New UI’s with better sensitivity to the user & the user’s physical environs • Great potential to simplify & enhance the UI •  new behaviors and services that users find compelling, useful, engaging, respectful • Design, Implementation, & Usability challenges… • Design must handle false positive / false negative cases • Not a panacea. Only seems helpful for some tasks. • Making things simpler vs. Loss of explicit control • Promising area that needs more work

21. Proximity Sensor • IR LED, 60o, 40kHz • IR receiver (used in TV’s) • Gaindistance • Some light sensitivity, e.g. sunlight

22. Touch Sensor • Mainly so we know whenuser is holding device • Useful to disambiguate “intentional” gestures from accidental ones • holding and tilting, vs. • tilted while sitting in briefcase • Also experimented with touch buttonsaround screen bezel

23. Software Architecture • “Context Server” whiteboard • Shared memory to read variables • Or, ask for Windows messages to notify of changes • Apps can post any synthesized info back to server • Some example context variables (events): • Holding, Duration • TiltAngleLR / FB • DisplayOrientation • Walking • Proximity (z) • ProximityState • Scrolling • VoiceMemoGesture

24. Touch bezel for explicit controlover tool bars Proximity of hand to screen? auxiliary UI appears when hand gets close Screen Real Estate Optimization