Input and Output Mobile and Physical Interaction, WS 2010/2011

1. Jörg Müller, Michael Nischt Hans-joerg.mueller@telekom.de Deutsche Telekom Laboratories, TU Berlin Input and OutputMobile and Physical Interaction, WS 2010/2011 The slides for this lecture are based on the lecture „Mobile and Physical Interaction“ by Michael Rohs and Georg Essl

2. Review • Organization • History of HCI • Some Research Topics • Exponential developments

3. Preview • Input and output modalities for mobile and physical interaction • Design space of input devices • Text input for mobile devices • Display output and input technologies • Haptics and audio output • Exercise Session • Simple Computer Vision with Processing and JMyron

4. The Design Space of Input Devices

5. Input Devices and Interaction Techniques • “An input device is a transducer from the physical properties of the world into logical parameters of an application” (Card et al.) • Interaction techniques combine input with feedback • Control processes generally need feedback loop

6. Input Devices • Design of human-machine dialogue = design of artificial languages • Input devices enable human-machine dialogues • Communicative intention  movements  application • Manipulations of input devices as sentences in a formal language • Primitive moves • Composition operators to generate sentences of moves

7. Properties of Input Devices • Property sensed (position, motion, force, rotation, etc.) • Absolute vs. relative sensing (position vs. movement) • Absolute sensing issue: nulling problem (physical position not in agreement with value set in software) • Relative sensing issue: clutching • Number of dimensions • 1D, 2D, 3D, 6D • Indirect vs. direct • Indirect: input space and output space are separate • Direct: input space = output space • Device acquisition time • Control-to-display (C:D) ratio

8. Language of Input Device Interaction • Modeling the language of input device interaction • Primitive movement vocabulary • Set of composition operators • Input device (M, In, S, R, Out, W) • M: manipulation operator • In: input domain • S: current state • R: resolution function(input-to-output mapping) • Out: output domain set • W: work properties(e.g. lag)

9. Language of Input Device Interaction • Modeling the language of input device interaction • Primitive movement vocabulary • Set of composition operators • Input device (M, In, S, R, Out, W) • M: manipulation operator • In: input domain • S: current state • R: resolution function(input-to-output mapping) • Out: output domain set • W: work properties(e.g. lag)

10. Motivation for Design Space • Need for systematization • Bring order to knowledge about input devices • “Design Space of Input Devices” (Card, et al.) • Framework for describing and analyzing input devices • Group input devices into families • Find gaps in the design space • Morphological design space analysis • Morphé: shape or form (Greek) • Morphology: the study of the form and structure of things • Input device designs as points in a design space

11. Generating the Design Space • Primitive movement vocabulary • Composition operators • Merge composition: cross product • Layout composition: collocation • Connect composition: output  input • Design space of input devices • Possible combinations of composition operators with the primitive vocabulary

12. The Design Space of Input Devices • Set of possible combinations of composition operators with the primitive vocabulary Merge Layout Connect • Touch screen? • Keyboard? • Trackball? • Isometric joystick?

13. The Design Space of Input Devices • More input devices • absolute position heavily populated • relative force and relative torque are empty • Touch screen • absolution position (X, Y) • Keyboard • absolute position (Z) • Trackball • relative rotation (rX, rY) • Isometric joystick • absolute force (F)

14. Testing Points in the Design Space • Use the space to evaluate devices • Expressiveness • “The input conveys exactly and only the intended meaning” • Problematic if Out  In do not match • Out  In: can input illegal values • Out  In: cannot input all legal values • Cf. 3D position with a mouse • Effectiveness • “The input conveys the intended meaning with felicity” • Pointing speed: device might be slower than unaided hand • Pointing precision: convenient selection of small targets • Errors • Time to learn • Time to grasp the device

15. Bandwidth • Speed of use depends on • Human: bandwidth of muscle group to which input device attaches • Application: precision requirements of the task • Device: effective bandwidth of input device

16. Mobile Text Entry

17. Text Entry on Mobile Devices • Mobile text entry • SMS (>1 billion SMS messages sent each day) • Email, calendars, etc. • Small devices require alternative input methods • Smaller keyboards, stylus input, finger input, gestures • Many text entry methods exist • Companies are ambitiously searching for improvements Key-based Finger-based Stylus-based Tilt-based

18. Text Entry Speed on Mobile Devices • Goal: High-speed entry at low error rates • Movement minimization • Low attention demand • Low cognitive demand • Entry speeds depend on task type and practice • Typical text entry speeds • Desktop touch typing: 60+ words per minute (wpm) • Soft keyboards: 40+ wpm after lots of practice, typically 18-28 wpm for qwerty, 5-7 wpm for unfamiliar layout • Handwriting speeds: 13-22 wpm

19. Keyboard Layouts for Mobile Devices • Qwerty variations • Designed to be slow • Prevent typing machines from jamming • alternate between sides of the keyboard

20. Home row Dvorak Keyboard • Speed typing by • Maximizing home row (where fingers rest) • Alternate hand typing • Most frequent letters and digraphs easiest to type

21. Fitaly and Opti Keyboards • Designed for stylus input on soft (on-screen) keyboards • Minimizing stylus movement during text entry • Stylus movement for entering the ten most and least frequent digrams:

22. Half-Qwerty and ABC Keyboards • Half-qwerty • One-handed operation • 30 wpm • ABC keyboards • Familiar arrangement • Non-qwerty shape

23. Very Small Devices • 5 keys (e.g., pager) • 3 keys (e.g., watch)

24. 1? ambiguity continuum Keyboards and Ambiguity • Keyboard miniaturization: Smaller keys, Less keys • Unambiguous keyboards • One key, one character • Ambiguous keyboards • One key, many characters • Disambiguation methods (manually driven, semiautomatic) 3 5 12 >26 keys

25. R U N N E R S U M M E R S T O N E S Ambiguity • Ambiguity occurs if fewer keys than symbols in the language • Disambiguation needed to select intended letter from possibilities • Typical example: Phone keypad ?

26. Unambiguous Keyboards • One key, one character • FasTap keyboard • Keys in space between keys • 9.3 wpm FastTap

27. Ambiguous Keyboards • One key, many characters • Standard 12-button phone keyboard, larger variants Nokia N73 Twiddler, chord keyboard Blackberry 7100

28. Manual Disambiguation • Consecutive disambiguation • Press key, then disambiguate (multitap) • Concurrent disambiguation • Disambiguate while pressing key (via tilting or chord) • Multitap • More keystrokes • Disambiguating presses on same key (timeout or timeout kill) • Tilting • Tilt in a certain direction while pressing • Chord-keyboard on rear of device • Not picked up by device manufacturers

29. Disambiguation by Multitap

30. Partridge: TiltType • Text input method for watches or pagers • Press and hold button while tilting device • 9 tilting directions (corners + edges) • Buttons select to character set • Kurt Partridge et al.: TiltType: Acceler-ometer-Supported Text Entry for Very Small Devices. UIST 2002 technote portolano.cs.washington.edu/projects/tilttype

31. Dictionary-Based Disambiguation (T9)

32. Simplified Handwriting: Unistroke • Single-stroke handwriting recognition • Each letter is a single stroke, simple recognition • Users have to learn the strokes • “Graffiti” intuitive unistroke alphabet (5 min practice: 97% accuracy) • Slow (15 wpm) • Users have to attend to and respond to recognition process • Recognition constrains variability of writing styles • Requires considerable processing power

33. Unipad: Language-Based Acceleration for Unistroke • Speeding up stylus-based text entry • Visual feedback required for handwriting • Eyes-free entry possible for unistroke • Look at suggestions during eyes-free unistrokes • Language-based acceleration techniques • Word completion list based on corpus (word, frequency) • Tap candidate • Suffix completion based on suffix list (“ing”, “ness”, “ly”, etc.) • Top-left to bottom-right stroke, tap suffix • Frequent word prompting (“for“, "the", "you", "and", etc.) • Tap frequent word MacKenzie, Chen, Oniszczak: Unipad: Single-stroke text entry with language-based acceleration. NordiCHI 2006.

34. Unipad: Acceleration by Word Completion • Word completion example • User is entering “hours” • State after two strokes (“ho”) • Experimental interface • First line shows text to enter • Second line shows text already entered • Pad below • Entering strokes • Word completion list MacKenzie, Chen, Oniszczak: Unipad: Single-stroke text entry with language-based acceleration. NordiCHI 2006. http://www.yorku.ca/mack/nordichi2006.html

35. Unipad: Acceleration by Frequent Word • Frequent word example • User is about to enter “of” • Pad shows frequent word list • User taps “of” MacKenzie, Chen, Oniszczak: Unipad: Single-stroke text entry with language-based acceleration. NordiCHI 2006. http://www.yorku.ca/mack/nordichi2006.html

36. Unipad: Acceleration by Suffix Completion • Suffix completion example • User is entering “parking” • State after 3 strokes (“par”) • Pad shows word completion list (incl. “park”) • User enters top-left to bottom-right stroke on “park” • Pad shows suffix list • User taps “ing” MacKenzie, Chen, Oniszczak: Unipad: Single-stroke text entry with language-based acceleration. NordiCHI 2006. http://www.yorku.ca/mack/nordichi2006.html

37. Unipad: Performance • Entry speed >40 wpm possible • KSPC ≈ 0.5 (key strokes per character) • Expert performance simulated on sentence“the quick brown fox jumps over the lazy dog” (43 chars) (27 strokes) MacKenzie, Chen, Oniszczak: Unipad: Single-stroke text entry with language-based acceleration. NordiCHI 2006. http://www.yorku.ca/mack/nordichi2006.html

38. EdgeWrite • Provide physical constraints • Moving stylus along edges and diagonals of square input area • People with motor impairments • Input = Sequence of visited corners • Example: Digits Wobbrock, Myers, Kembel: EdgeWrite: a stylus-based text entry method designed for high accuracy and stability of motion. UIST'03. http://depts.washington.edu/ewrite/

39. QuickWriting: Gesture-Based Input • Combine visual keyboards with stylus movements • Motor memory for paths • Following a path through letters of the word to enter • Reduced stress and fatigue compared to continuous tapping • Ken Perlin: Quikwriting: Continuous Stylus-based Text Entry. UIST’98. Quickwriting, http://mrl.nyu.edu/~perlin/demos/quikwriting.html

40. Display and Touch Screen Technologies

41. Light Emitting Diodes (LED) • Cheap • Fairly high power consumptionwhen on • Use outside of visual range • Infrared • Ultraviolet http://www.adobe.com/de/designcenter/thinktank/livingskins/images/Living_Skins_fig04.jpg

42. Liquid Crystal Display (LCD) • An LCD cell is a voltage-controlled “light valve” • Twisted nematic effect • Liquid crystal molecules form helix structure between electrodes • Electric field aligns molecules (controls amount of twisting) • Orientation of molecules controls orientation of polarized light LCD pattern Off state On state

43. Plasma Display • Like a flourescent lamp, emit light • UV light hits phosphor, creates visible light • Very large implementations • High power consumption (depends on image)

44. Organic Light Emitting Diode (OLED) • Emit light (like LED) • Very thin, bendable, can be printed on any foil • High contrast • Aging effect

45. E-Paper • Reflective display (the more sun shining on it, the better) • Only consumes power while changing content • No large displays available yet • Slow update rate

46. Projection • Back- or Frontprojection • Robust and cheap for large displays • Difficult with ambient lighting • LCD, DLP, LED (uses DLP), LCoS (like LCD),

47. Laser Projection • Nice for large distances • High contrast • Expensive • May be dangerous

48. Touch Screens • Resistive • Capacitive • iPhone • Surface Acoustic Wave (SAW) • Light Array • Frustrated Total Internal Reflection • Jeff Han’s multitouch table • Diffuse Illumination

49. Polyester Film Upper Resistive Circuit Layer Conductive ITO (Transparent Metal Coating) Lower Resistive Circuit Layer Insulating Dots Glass/Acrylic Substrate Touching the overlay surface causes the (2) Upper Resistive Circuit Layer to contact the (4) Lower Resistive Circuit Layer, producing a circuit switch from the activated area. The touchscreen controller gets the alternating voltages between the (7) two circuit layers and converts them into the digital X and Y coordinates of the activated area. Resistive Touch Screens (www.fastpoint.com)

50. Senses capacitive changes Only works with finger, not with stylus iPhone Uses additional grid for better multitouch disambiguation Capacitive Touch Screens (www.unwiredview.com)