1 / 75

Automatically Generating High-Quality User Interfaces for Appliances

Automatically Generating High-Quality User Interfaces for Appliances. Jeffrey Nichols Thesis Proposal Presentation Thesis Committee: Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr. Appliances are complex and their user interfaces are often hard to use!. Problem, cont.

mea
Download Presentation

Automatically Generating High-Quality User Interfaces for Appliances

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Thesis Proposal Presentation Thesis Committee: Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr.

  2. Appliances are complex and their user interfaces are often hard to use! Problem, cont.

  3. Problem, cont.

  4. Problem, cont. May 2003 CNN – Jan 2004 April 29, 1991

  5. Thesis Statement “ A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the user’s performance is better than with the manufacturers’ interfaces for the appliances.”

  6. Approach: Use Mobile Devices Common, cheap, and often have ability to communicate 10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers

  7. Approach, cont. Specifications Control Feedback Use mobile devices to control all appliances in the environment Mobile Devices Appliances Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

  8. Automatic Generation of UIs Benefits • Multiple modalities (GUI + Speech UI) • All interfaces consistent for a user • With conventions of the handheld • Even from multiple manufacturers Addresses hotel alarm clock problem • Create user interfaces that control multiple connected appliances e.g. a home theater

  9. Important Work By Others INCITS/V2 Standardization Effort Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002] Controlling Appliances • Xweb [Olsen Jr., UIST 2000] • Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] • Speakeasy [Newman, UIST 2002] Model-based User Interface Work • UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993] • MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995] • eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002] • User Interface Modeling Language (UIML) [Abrams, W3C 1999] None of these systems have been validated to generate high-quality interfaces without designer intervention.

  10. Research Approach • Hand-design remote control interfaces • Determine functional information needed from appliances to design user interfaces • Design language for describing appliance functions • Build interface generators for multiple platforms • Perform user studies to evaluate the interface generators

  11. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

  12. Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand • AIWA Shelf Stereo • AT&T Telephone/Answering Machine Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information

  13. Hand-Designed Interfaces Interfaces for Microsoft’s PocketPC (simulated remote control) telephone stereo

  14. Comparison Study Compared performance of first-time users (not experts) Procedure • Each subject worked two sets of tasks both stereo and phone controlled for order and interface Performance Metrics • Time to complete all tasks • Number of times a user manual was needed • Number of missteps

  15. Comparison Study Results Using our interfaces, users were twice as fastand made half as many errors All differences are significant (p < 0.05)

  16. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

  17. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

  18. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

  19. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

  20. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

  21. Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

  22. Language Design Informed by hand-designed interfaces • What functional information was needed to create interfaces? Additional Requirements • Support complete functionality of appliance • No specific layout information • Only one way to specify anything Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

  23. Language Elements Elements • State variables & commands • Labels • Group tree • Dependency information Example media player specification • Play, stop, pause, next track, previous track • Play list

  24. Language Elements, cont. State Variables and Commands • Represent functions of appliance • State variables have types • Boolean, Enumeration, Integer, String, etc. • Variables sufficient for most functions but not all • e.g. “seek” button on a Radio

  25. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

  26. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

  27. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

  28. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

  29. Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) • Allows feedback to user when a function is not available

  30. Interface Generators Generators for Two Modalities Graphical • Desktop, PocketPC, and Microsoft Smartphone Speech • Ph.D. work of Thomas Harris • Built on top of the PUC framework • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

  31. Graphical Interface Generator Rule-based approach • Multiple phases that iteratively transform a specification into a user interface Focuses on panel structure of user interface • Small groups of controls have basic layouts • Complexity comes from structure of groups • Structure can be inferred from dependency info!

  32. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

  33. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

  34. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

  35. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

  36. Generation Process • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

  37. Generation Process With layout fixing rules Without layout fixing rules • Determine conceptual layout • Infer panel structure from dependencies using “mutual exclusion” property • Choose controls (decision tree) • Choose row layout (one column, two column, etc.) • Allocate space • Examine panel contents and choose sizes • Instantiate and place controls • Fix layout problems

  38. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

  39. High-Level Conventions Problem • Human designers rely partly on conventions when making an interface • Users expect their appliances to use conventions they know about

  40. Smart Templates Need a way to specify high-level information to interface generators Solution • Mark groups with tags that identify high-level information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect

  41. Smart Templates, cont. Features • Parameterized • Specified using primitive elements of specification language

  42. Preliminary Implementation A few templates: image, image-list, media-controls, time-duration Interfaces with Smart Templates

  43. Continuing Work Define and implement Smart Templates • date, mute, power, time-absolute, volume, etc. • Develop more as more appliances are specified Combinations of templates • Less implementation cost than a new template • e.g. date and time-absolute Use of templates with data already on controller device • e.g. calendar and address information • Might allow user to enter address from contact list into navigation system

  44. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

  45. Interface Consistency PUC devices have a unique opportunity to provide consistency for the user • Personal device • Used for interacting with most appliances Two ways that PUC UIs can be made consistent • With other applications on the same device • With past interfaces for similar appliances

  46. Consistency with Past Interfaces Two sub-problems to address: • Similarity Which functions of a new appliance are similar to the functions of interfaces generated in the past? • Consistency For similar functions, what rules from previous interfaces can be applied to ensure consistency?

  47. Similarity Problem Difficult to conclusively know whether two functions from different appliances are the same • Very little semantic information in the specification language Can estimate similarity based on properties of state variables • Smart Template • Name • Group Name • Labels • Type Improve estimate by looking at relationships between multiple similar variables

  48. Similarity & Consistency Problems new previous sparse similarity new previous new previous branch similarity significant similarity

  49. Outline • Introduction • Hand-Designed Interfaces and User Studies • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Proposed Contributions & Schedule

More Related