1 / 57

Interaction Techniques with Mobile Devices

Interaction Techniques with Mobile Devices. Jingtao Wang jingtaow@cs.berkeley.edu. March 6th, 2006 Guest Lecture for CS160. Agenda. Why Mobile Devices Matters Ubiquitous Computing Key Challenges in Designing Mobile Applications

evan
Download Presentation

Interaction Techniques with Mobile Devices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Interaction Techniques with Mobile Devices Jingtao Wang jingtaow@cs.berkeley.edu March 6th, 2006 Guest Lecture for CS160

  2. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

  3. Why Mobile Devices Matters • 6.5 billion people in the world • 1.5 billion cell phones worldwide • 500 million PCs (?) • 46 million PDAs • 1 million TabletPCs • Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned

  4. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

  5. Mark Weiser (1952 – 1999) Introduced the idea of “ubiquitous computing”

  6. Weiser’s Vision “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it”

  7. Weiser’s Vision: Pervasive • Mainframe • many people share 1 computer • PC • 1 person with 1 computer • Ubicomp • many computers server each person

  8. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

  9. Key Challenges in Making Mobile Applications • Limited Physical Resources • CPU, Memory, Screen Size, Input Devices, Battery Life etc • Diversified Context of Use • Different Activities • Limited Attention

  10. Limited Physical Resources • A mobile device usually has 1/100 CPU power, 1/30 Screen resources, 1/20 Memory, and extremely limited input devices when compared with desktops in the same era. • Small Screen Geography is different b. Small Screen a. Large Screen

  11. Diversified Context of Use

  12. Different Activities • People use small-screen devices for different activities than desktops; don’t assume you understand these activities already

  13. Limited Attention • Don’t assume your applications have people’s full attention; they’re doing something else while using your device.

  14. Context, Activity, Attention • There is more opportunity for purpose-specific or context-specific devices than for general-purpose solutions that try to work for everyone in any situation.

  15. One Sentence Summary • There is no silver bullet in designing mobile applications, but there is one sentence you should remember - Mobilize, Don’t Miniaturize !

  16. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices

  17. Input Techniques for Mobile Devices • Pointing • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

  18. Common Pointing/Navigation Techniques JogDial iPod Dialpad TrackPoint Touch Screen Four-directional keypad

  19. TinyMotion – Camera Phone Based Pointing • Detecting the movements of cell phones in real time by analyzing image sequences captured by the built-in camera. • Typical movements include - horizontal and vertical movements, rotational movements and tilt movements.

  20. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

  21. (Virtual) Keyboard Input

  22. How to Make QWERTY Keyboards Portable ? • Break

  23. Making QWERTY Keyboards Portable • Reducing the Absolute Size • Reducing the Number of Keys • Making Keyboards Foldable • Projective Keyboard

  24. Projective Keyboard From http://www.vkb-tech.com

  25. Projective Keyboard – Working Mechanisms • Template creation • Reference plane illumination • Map reflection coordinates • Interpretation and communication

  26. Can We Perform Better Than QWERTY? • Originally QWERTY layout is manually optimized for two handed, alternative typing to minimize mechanical jam OPTI ATOMIK FITALY OPTI II

  27. Theories Behind Quantitative Keyboard Layout Optimization • Fitt’s Law • Digraph Distribution Model in a Language

  28. Can We Use Less Buttons than a Full QWERTY? 12-button Keypad 15-button Keyboard Half Keyboard

  29. Disambiguation Methods for Reduced Size Keyboard • The QWERTY keyboard itself is ambiguous! ( A vs. a, 3 vs. #) • Pressing Several Keys together (shift key) • Multiple Key Press • Multi-Tap (22.5 wpm*) • Two-Key Input (25.0wpm*) • Dictionary/Statistics Based Disambiguation Methods • T9/T15 (45.7 wpm*) • LetterWise *Performance Upper Bound Estimation from Silfverberg 2000

  30. FingerSense – Button Disambiguation by Fingertip Identification • Differentiating a pressing action by identifying the actual finger involved • Can be Faster than Regular Tapping When the Adjacent Tapping Involves Different Fingers and Different Buttons (59% on a phone keypad)

  31. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

  32. Handwriting Input • 1938 George Hansel, U.S. Patent 2,143,875, machine recognition of handwriting • 1957 T. L. Dimond's stylator - the first on-line handwriting recognizer prototype • Newton, Palm Pilot, PocketPC, CrossPad, TabletPC

  33. Handwriting Recognition - Terminology • Printed Character Recognition (OCR) • Relatively mature these days, key challenges – layout analysis, fonts recovery, robust recognition for low quality, low resolution input • Major Usage – Digital Library • Handwritten Character Recognition • Online HWR (With Temporal info) • Character, Word, Sentence Level • Offline HWR (Using raster image as input, no temporal info) • Major Usage : Bank Check Recognition, Postal Automation

  34. Word/Sentence Level Recognizer • Build on Top of Character Recognizer • General Strategy • Over Segmentation • Call Character/Component Recognizer, Get a List of Candidates with Scores • Apply Geometry Spatial Information ( size, component gap ) Language Information (Dictionary, Language Model etc) to Each Sub Path • Use Hypnosis Search (Dynamic Programming, A* etc) to Determine the Best Possible Path

  35. Challenges in Online HandwritingRecognition • Character Set/Dictionary Size (Especially Asian Languages!) • Cursive Writing Styles/Broken Strokes/Duplicate Strokes/Omitted Components • Stroke Order Variations • Limited memory and CPU Power in Small Devices

  36. Some Prototype Recognizers from IBM

  37. New Form Factors - Anoto Pen • Commercial Product is Available In the U.S. Market (Logitech IO Pen) • Uses A Camera Mounted Beside the Tip of the Pen and Preprinted Dot Patterns to Detect Pen Movment

  38. SHARK – Shorthand Writing on Stylus Keyboard • A Combination of Virtual Keyboard and Handwriting Recognition • Writing Shape of a Word (Shorthand) is Defined By the on Screen Location of Characters in the Word

  39. EdgeWrite Input • An EdgeWrite user enters text by traversing the edges and diagonals of a square hole imposed over the usual text input area • Faster and More Reliable Than Regular Graffiti • Especial Useful for People with Motor and Muscle Disabilities

  40. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

  41. Input Techniques for Mobile Devices • Pointing • Sensor Augmented Input • Text Input • (Virtual) Keyboard Input • Handwriting Input • Speech Input • Marker Based Input

  42. Emerging Marker Based Interactions on Camera Phones

  43. Towards More Sensitive Mobile Devices

  44. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

  45. Peephole Displays (With Demo)

  46. Zoomable Interface on Mobile Devices • ZoneZoom By Microsoft • Take advantage of spatial memory VS.

  47. Halo - A Virtual Periphery for Mobile Devices Provding Visual Cue for Objects Located Out of the Small Screen

  48. Agenda • Why Mobile Devices Matters • Ubiquitous Computing • Key Challenges in Designing Mobile Applications • Input Techniques for Mobile Devices • Output Techniques for Mobile Devices • Interact With Other Devices

  49. Using Mobile Devices with Desktop Computers • Pebbles Project at CMU • Using a PDA as additional keypad, touch pad, scroll wheel and controller of PointPoint slides for desktop Applications • http://www.pebbles.hcii.cmu.edu/

  50. Using Mobile Devices with Laptops Wang 2002

More Related