Visions for Mobile Devices Beyond Their Current Role

1. Visions for Mobile Devices Beyond Their Current Role Brad A. Myers Carnegie Mellon University Human Computer Interaction Institute bam@cs.cmu.edu http://www.bam.hcii.cmu.edu/

2. Our Research on Handhelds, 1 • Pebbles project • Since 1997 • Presented at Faculty Summit 2002, 2003, 2004 • http://www.pebbles.hcii.cmu.edu • How handhelds can be used together with desktop computers • Remote Commander • Shortcutter • SlideShow Commander

3. Our Research on Handhelds, 2 • Pebbles Personal Universal Controller (PUC) • Automatically create remote control panels on PocketPC and Phone for appliances • Appliance description in XML • Interfaces that are consistent for the user • Automatically combine interfaces for multiple appliances

4. Thoughts on Research Topics for the Future of Handhelds

5. Data Integration • Multiple antennas in all devices • Cellphone + BlueTooth + WiFi • Ability to be constantly connected • Not just once a day for synchronization • Can re-think the data model • Integration of all of the user’s data • Small to large pieces of data • Cut-and-paste across devices • Making short notes and short annotations (meta-notes) a fundamental part of data that is stored • Intelligent and automatic consistency of data • Never re-enter data more than once anywhere!

6. Where is the Data? • On desktop, “in the network”, on body, synchronized everywhere? • Storage is basically free but still have the synchronization problem • Could put on network, but then loose reliability, worry about security/theft • Worry if lose mobile device with data • Make control of what is shared with others easier than emailing

7. Application Integration • Portable applications across differentform factors • Run same application on handheld as on PC? • Web pages • Word  Pocket Word • Automatic adaptation to I/O capabilities • Even dynamically as application runs • Would help with innovating in hardware since might just work on the new device

8. Entering Data • Text entry still hard • Hand-helds have voice recorders, cameras, styluses for strokes and handwriting • Easier on a phone to take a picture than to type • Rich data types input and output • Need better ability to handle all these types as easily as text • Handwriting = picture = typed = spoken • Anywhere can use one can use another • Pictures as the answer to questions • Use context, location, etc. so enter less information • Intelligent interfaces to fill in field with right values? • Use history, etc. to learn from user

9. Handheld and PC Augment Each Other When Nearby • Touch screen, camera, voice recorder, phone modem, as PC’s I/O • PC’s keyboard, mouse, big screen for handheld’s I/O • Opportunistic annexing of anything in the environment (displays, keyboards, antennas, networks, etc.) • Office, conference room, home, automobile • E.g., Use car’s speakers, microphones, heads-up or nav-system display • http://www.cc.gatech.edu/projects/PIE/projects/oppannex.shtml • UI research issues: where best for: • Control: across appliances/devices, input methods • Output: displays on TV vs. front panel vs. handheld

10. Form Factors For Handhelds • Fundamental human sizes: • In Hand • Phone, PDA -> smaller, pocket • On Arm • Tablet • On Wrist • Watch • On Tables • Clamshell Windows CE  not successful • New Mini-pcs • Laptop • Wearables • Sensors like BodyMedia • Tradeoffs: portability, battery life, screen size (readability), text entry

11. Special Purpose Devices vs. Convergence • Mobile game device, PDA, phone, pager, still camera, video camera, music player, movie player, watch, flash memory, … • Star Trek: Tricorder, Communicator, Phaser, Medical scanner, portable memories • Or one device that does it all? • Marketing issues – will people create content (games, apps, movies) for your device? • HCI issues • Good game control UI ≠ good phone control • Larger display ≠ smaller device • Battery life

12. Convergence vs. Special Purpose, cont. • If many devices, still shouldadhere to “enter once” principle • Scattered around environment,vs. carry around in pocket/on body

13. Idea: Disassemble the Parts • Each function in separate device • Speaker in ear, microphone, display on arm, processor in pocket, memory, antennas • Integrate with watch display, input • E.g., Intel “personal server” • Addresses the one v. many issue? • Specialized I/O mechanisms (sold separately?) attach to common core • Game controller/game media reader • Camera lens

14. Other Research Areas for Handhelds • New UIs for Handhelds • Not just shrunk desktops • More uses for cameras • Handicapped access • Digital inclusion using handhelds • Security in general

15. Summary • Still lots of interesting and important research topics in the handheld / mobile area! • Challenge: How can handheld devices improve the user interfaces of everything else, and not just be another gadget to be learned • 6.5 billion people in the world • 1.5 billion cell phones worldwide, vs. 500 million PCs (?)vs. only 46 million PDAsvs. 1 million TabletPCs

16. Thank You! Visions for Mobile Devices Beyond Their Current Role Brad A. Myers Carnegie Mellon University Human Computer Interaction Institute bam@cs.cmu.edu http://www.bam.hcii.cmu.edu/

18. -- Extras / Not Used --

19. Handhelds as Platform When Can’t Afford a PC • Less-developed countries • How use mobile phones for what we would use a PC for? • What makes sense to try to do? • Already bought a cell phone, computation is free, what can be done with it? • Computers used for: • Communication, entertainment, business stuff, … • Collaboration through handheld as a way to bring experts to less-developed countries • Applications in Health care, Education, Agriculture, etc. • Specialized applications  how create apps. cheaper? Easier to program/customize

20. Text Entry on Handhelds Still Hard • Lots of research, still not solved EdgeWrite QuikWriting Xnav Shark Dasher

21. More Uses for Camera • Scanner for text, UPC reader, sign translation • Sensor for movement, • Mouse replacement • Gesture recognition • Where am I? (augment GPS) • Games that use camera as a sensor • Face recognition • Plant recognition, disease recognition, … • Type on table with camera detecting virtual keys • Collaboration support: show the objects discussed • But needs much better integration  still too hard to send/receive pictures

22. New User Interface Metaphor? • Completely new UI metaphor that is appropriate for phones/mobile devices • Maybe not just applying the 25-year-old desktop UI to the phone • What could be done? • Haptic interfaces for mobile devices • Besides just vibrate • More appropriate than for desktop

23. Other Research Issues • Handicapped access • Blind: Braille handhelds?, talking • Muscular difficulties • Cognitive difficulties • Security in general • Safety of data • Trustedness of devices • Transmission of data