1 / 44

Lecture 17: Development of and for Devices

Lecture 17: Development of and for Devices. Jake Wobbrock 05-830 Advanced User Interface Software. Not a Proper Toolkit Lecture. This lecture might have been about UI toolkits for devices There’s just one problem There really aren’t any

robertellis
Download Presentation

Lecture 17: Development of and for Devices

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Lecture 17:Development of and for Devices Jake Wobbrock 05-830Advanced User Interface Software

  2. Not a Proper Toolkit Lecture • This lecture might have been about UI toolkits for devices • There’s just one problem • There really aren’t any • At least, not like we think of toolkits for desktop software • Instead we will talk about “things that help you develop for devices.” • And other interesting device issues…

  3. Overview • Introduce work in multiple semi-related areas of software development of and for devices • Palm OS • Pocket PC / Windows CE • .NET Compact Framework • J2ME • Waba • BREW • WAP and WML • DirectX

  4. Overview cont. • Highlight HCI research projects with and for devices • Phidgets • Rapid prototyping for physical devices (physical widgets) • Pebbles Project • Interoperation of handhelds and PCs • Handheld Web browsing (various efforts) • WebThumb, Power Browser, The Gateway, M-Links

  5. Motivation: Handhelds • Cell phones and PDAs have seen huge growth in use over recent years. • “Last year [2002] there was one cell phone sold for every 15 human beings on the planet … Worldwide mobile phone unit sales totaled 423.4M in 2002, a 6% increase from 2001.” • “From 1999 to 2000 the number of PDAs sold nearly doubled, from some 3.6M units to 6.9M… By 2004, the number of units is expected to top 33.5M.” Source: Biz Journals

  6. Motivation cont.: Gaming • Nintendo GameBoy • NTT DoCoMo Inc. handhelds running Java • Others by Qualcomm, L.M. Ericsson, Motorola, Siemens … • Convergence: Games for phones and PDAs, not only custom handhelds Source: CNN

  7. The Punch Line • PDAs, phones, other handhelds experiencing great growth in use • Development community needs to have good tools and toolkits • What is available today? • Sadly, not much (at least compared to what’s available for the desktop)

  8. Palm OS Visit http://www.palmsource.com/products/ • Palm OS by PalmSource • Acer • AlphaSmart • Fossil • Garmin • Handspring • HandEra • Kyocera • Palm • Samsung • Sony • Symbol

  9. Palm OS Facts • As of January 2002: • 16M Palm-powered PDAs shipped (to date) • 7000 applications • 150,000 registered developers • Growing at 1000 per month Source: Palm OS Programming, O’Reilly

  10. Palm OS Development • Two main ways • CodeWarrior for Palm OS • PRC-Tools (GCC) • Other ways • Sun KVM and J2ME • IBM VisualAge Micro Edition (Windows or Linux) • Jump (use Java, compiles to Motorola 68K) • Waba (subset of Java, stripped-down VM) • PocketStudio (Pascal-based, aimed at Delphi users) • Pocket C (develop on the handheld for the handheld) • Assembler SDK (ASDK) – write directly for Motorola 68K • Pocket Smalltalk (includes garbage-collecting VM)

  11. CodeWarrior for Palm OS • Special line of Metrowerks CodeWarrior releases for Palm OS development • Three key components • Editor • Constructor • Palm OS Emulator (POSE) • Write code in C/C++ • Downloaded to Palm via HotSync™

  12. Editor

  13. Constructor

  14. Runs on the desktop Emulates from a “very low level” Designed to simulate speed despite running on a desktop Useful for debugging Not useful for interactions requiring pen-dexterity Emulator (POSE)

  15. Technical Details • Palm OS apps center around event cascade • Catch events by defining handlers and handling them • But not nearly as much framework as Visual Basic or MFC… more like Win32 • Often involves large switch statements • Single-threaded, single-process

  16. The Event Cascade

  17. CodeWarrior Summary • Generally the best option • Originally for Macintosh, then for Windows! • BUT… • Constructor is often not as WYSIWYG or Direct Manipulation as you’d like (e.g., text fields, scroll bars) • It lacks options for all programmatically-settable options that affect visual appearance of controls (e.g., button borders) • Emulator doesn’t work exactly like the device • Sometimes this matters, often this doesn’t

  18. PRC-Tools (GCC) • Alternate means for Palm OS development • Developed before CodeWarrior ran on Windows by the Free Software Foundation • Collection of tools for Unix or Windows • Gnu C Compiler (GCC) for Motorola 68K • Build-PRC (combine binaries into *.prc file) • Gnu Debugger (GDB) • PilRC (compiles resource files) • Falch IDE combines these with a UI

  19. Pocket PC? Windows CE? • Pocket PC • Not considered (strictly-speaking) a PDA • Or at least not marketed that way • Meant to be a full-fledged PC but pocket-sized • Refers to a device (platform) • Also to a set of standard applications common on all Pocket PC devices, esp. Pocket PC Shell • Windows CE (WinCE) • Refers to just the OS that runs on Pocket PCs • Has large number of capabilities (components) for the Pocket PC developer to choose from • Current version is WinCE .NET

  20. Pocket PCs vs. Palm OS Devices • “Basic PDAs allow you to store and retrieve addresses and phone numbers, maintain a calendar, and create to-do lists and notes. More sophisticated PDAs can run word processing, spreadsheet, money manager, games and electronic book reading programs and also provide email and Internet access.” Source: Microsoft

  21. Pocket PCs • Toshiba • Compaq iPaq • HP Jornada • Audiovox • T-Mobile • Dell Axim • ViewSonic • Razor Zayo • Cassio Cassiopeia • NEC MobilePro

  22. Pocket PC Facts • Projected 2003 sales between 14M – 17M devices • Compare to PC sales of 135M – 138M • Pocket PC projected to catch up by 2008! • Sales hit 10M after just 10 months on the market • Introduced April 2000 • In May 2001, held 26% of market. Rest held by Palm OS devices. Source: C|Net

  23. The Shrunken Desktop • Pocket Word • Pocket Excel • Pocket Outlook • Pocket IE • Pocket Windows Media Player • … ??

  24. Pocket PC Development • eMbedded Visual Tools 3.0 (VC++, VB) • Largely just like Windows programming • Can even use MFC for VC++ • Compiles for many different processors

  25. Challenge • Problem: Portability • Not all Pocket PCs utilize the same WinCE components • So not all Pocket PC apps will run on all Pocket PCs • Despite same underlying OS! • Example: The menu bar at the bottom is specific to Pocket PCs, not WinCE • Created and passed to the Pocket PC shell • Developers often have to change much UI code for different Pocket PC platforms • Example: Handheld PC (H/PC) is wide and short, so tall dialogs have to be re-laid-out

  26. Resource Editor

  27. Pocket PC Summary • Powerful palmtop computers • More complicated than Palm OS Devices (e.g., can be multi-threaded) • Less integrated vertically than Palm OS devices • Complicates development (e.g., UI components not often supported on all Pocket PC devices) • With more power and capability comes more complexity and complication • Affects the toolkits for development on such devices

  28. .NET Compact Framework • Subset of .NET framework for desktops • Not PDA-specific but targeted to any “smart devices”: PDAs, mobile phones, set-top boxes, automobiles, etc. • Delivered as extensions to Visual Studio .NET called “Smart Device Programmability” • Tools and programming model are same as for desktop .NET platform • Easy transition from .NET desktop development to .NET compact framework development

  29. .NET Compact Framework cont. • Argument is that there are currently too many mobile CPUs and OSes, so development is difficult • True – but we know the Microsoft answer • .NET would allow integration of devices across multiple platforms, networks, and programming languages • A vision behind .NET in general • Compact Framework is a subset of all .NET capabilities • Reduces footprint on resource-strained devices

  30. J2ME – Java 2 Micro Edition • Uses subset of Java Virtual Machine for smaller footprint (i.e., KVM) • Highly optimized runtime environment • Not PDA-specific: smart cards, pagers, mobile phones, set-top boxes, vehicle telematics systems (e.g., OnStar system) • Set of APIs defined by the Java Community Process Program™ • Includes user interface, security, networking protocols, and more

  31. J2ME Architecture • J2ME Architecture defines… • Configurations: A virtual machine and a minimal set of class libraries • For Palm OS, configuration is “Connected Limited Device Configuration” (CLDC) • Profiles: Higher-level APIs that further define the application, user interface, and device-specific properties • For Palm OS, profiles are PDA and “Mobile Information Device Profile” (MIDP)

  32. J2ME Architecture cont. From http://java.sun.com/j2me/j2me-ds.pdf

  33. Waba • Subset of Java • Stripped-down VM: small and fairly fast • No longs, doubles, exceptions, or threads • Open source: http://www.wabasoft.com/ • Waba defines… • A language (strict subset of Java) • A virtual machine (Waba VM) • A class file format (strict subset of Java bytecode) • A set of foundation classes

  34. BREW • Qualcomm’s Binary Runtime Environment for Wireless (BREW) • Like J2ME, provides application execution environment • Goes farther! Provides business model for certifying, downloading, and charging premium content • With J2ME, this is left entirely to the marketers and developers to figure out for themselves • Based on C++ (so it is fast, and small binaries)

  35. WAP and WML • WAP (Wireless Application Protocol) • Provides “flat” Web content on handhelds • Requires constant connectivity by the device • Lacks a rich interactive experience for end-users • WML (Wireless Markup Language) • Specifies content and user interface for WAP-delivered information • Based on XML: so describes data, not just presentation of data • WMLScript • WAP:WML:WMLScript :: HTTP:HTML:JavaScript • Provides only minimal interactivity

  36. WAP and WML cont. • Four things specified by WML • Text and image presentation and layout, including a variety of formatting commands • Deck/card organizational metaphors for information (similar to HyperCard) • Inter-card navigation and linking • Card parameterization and state management • The fate of WAP • Hasn’t been the success its advocates hoped • New push to make possible highly interactive content • J2ME, BREW, .netCF, Waba, but these all send a VM and increase footprint

  37. Game Development: DirectX • DirectX: Advanced suite of multimedia APIs • DirectDraw, Direct3D, DirectSound, DirectMusic, DirectInput, DirectPlay, DirectShow… • Allows accessing of special hardware capabilities (e.g., graphics and sound cards) without requiring hardware-specific code • DirectX exists for Pocket PC • Handheld gaming projected to be a big new market opportunity • Again, similar to programming for desktop

  38. Transition from (Mainly) Industrial Tools to Research Initiatives for and with Devices

  39. Phidgets (Greenberg et al.) • Phidgets are “physical widgets” • Play the same role in physical UIs as widgets do in graphical UIs • Package IO • Hide implementation details, provide abstraction • Have additional characteristics • Require a connection manager • A link between the software and hardware parts • A simulation mode in software only

  40. Phidget Architecture • Physical Device • Packaged physical unit to be used by the designer. Most phidgets are built around a circuit board. • Wire Protocol • Communication protocol between device and host computer. Not visible to end-programmers. • Phidget Manager • COM object that has event-based API for end-programmers for connection management. • Phidget-specific COM objects • Created by the Phidget Manager whenever a device is seen. They correspond 1:1 to physical devices. • IGlabPhidget interface • Generic interface which all phidgets have. End-programmers use it for basic identification and capabilities. • Phidget-specific interface • Capabilities specific to a particular phidget. Extends the basic IGlabPhidget interface. • Phidget ActiveX controls • Wraps up the phidget code and provides an on-screen user interface and the ability to simulate the phidget as an control.

  41. Pebbles (Myers et al.) • Research project exploring the interoperation of handhelds and PCs • Handhelds can be an aspect of the user, PCs can be a fixture of the environment • Users may move from environment to environment, PCs may stay behind • Handhelds can control other devices too • Personal Universal Controller • Handhelds can be used to control a PC by people with motor impairments

  42. A Current Research Issue • Handheld User Interfaces for Web Browsing • Challenges • Constrained screen-space • No mouse cursor (e.g., no link roll-overs) • Pages designed on a desktop for display on a desktop • Scrolling more difficult • Graphic refresh can be slow, scrolling blurry

  43. Handheld Web Browsing • The Gateway (MacKay) • Pages displayed as thumbnails expand as certain parts of them are interacted with (CHI 2003). • WebThumb (Wobbrock et al.) • Exploration of interaction techniques for better handheld browsing (UIST 2002). • M-Links (Trevor et al.) • Uses modes to separate action from navigation on mobile phone Web pages (UIST 2001). • Power Browser (Buyukkokten et al.) • Generates summary views of handheld web content for display (CHI 2000).

  44. Conclusions • Toolkits for development for devices are far from Amulet-level sophistication • Many processors and devices exist – complicates development and toolkit design • Numerous standards and niche technologies • Minimizing footprint on resource-constrained devices is a key constraint • HCI research can tackle issues beyond the specifics of device hardware and software • e.g., interaction techniques for PC/PDA interoperability • e.g., interaction techniques for browsing

More Related