Automatically Generating High-Quality User Interfaces for Appliances

1. Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Symposium Presentation Seventeenth Annual ACM Symposium on User Interface Software and Technology October 24, 2004

2. Appliances are complex and their user interfaces are often hard to use! Problem

3. Approach: Use Mobile Devices Built for general-purpose interaction Common, cheap, and often have ability to communicate 10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers in US

4. Approach, cont. Specifications Control Feedback Use mobile devices to control all appliances in the environment Mobile Devices Appliances Key Features Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation

5. Why Automatic Generation? Benefits • Multiple platforms and modalities (GUI + Speech UI) • All interfaces consistent for a user • With conventions of the handheld • Even from multiple manufacturers • Create user interfaces that control multiple connected appliances e.g. a home theater

6. Outline • Introduction • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Conclusion

7. Research Approach • Hand-design remote control interfaces • Determine functional information needed from appliances to design user interfaces • Design language for describing appliance functions • Build interface generators for multiple platforms • Perform user studies to evaluate the interface generators

8. Initial Approach What information is needed about the appliance to automatically generate remote control interfaces? Investigate via a design process Create interfaces by hand • AIWA Shelf Stereo • AT&T Telephone/Answering Machine Improve quality with heuristic analysis and think-aloud studies with several users Compare interfaces with actual appliance interfaces to validate PUC concept Analyze interfaces for functional information

9. Language Design Informed by hand-designed interfaces • What functional information was needed to create interfaces? Additional Requirements • Support complete functionality of appliance • No specific layout information • Only one way to specify anything Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/

10. Language Elements Elements • State variables & commands • Labels • Group tree • Dependency information Example media player specification • Play, stop, pause, next track, previous track • Play list

11. Language Elements, cont. State Variables and Commands • Represent functions of appliance • State variables have types • Boolean, Enumeration, Integer, String, etc. • Variables sufficient for most functions but not all • e.g. “seek” button on a Radio

12. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

13. Language Elements, cont. Label Information One label not suitable everywhere • The optimal label length changes with screen size • Speech interfaces may benefit from pronunciation and text-to-speech information “Label Dictionary” • A group of semantically similar labels • Different lengths • Information for different modalities

14. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

15. Language Elements, cont. Group Tree • Specify organization of functions • We use n-ary tree with variables or commands at leaves • Also used for specifying complex types Lists Unions

16. Language Elements, cont. Dependency Information • Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than, Is Defined Linked with logical operators (AND, OR) • Allows feedback to user when a function is not available

17. Interface Generators Generators for Two Modalities Graphical • Desktop, PocketPC, and Microsoft Smartphone Speech • Collaboration with others at Carnegie Mellon • Built on top of the PUC framework • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

18. The PUC System Architecture APPLIANCES (Stereo, Alarm Clock, etc.) PUC DEVICES (automatic interface generation) ADAPTOR (publishes description + appliance state + controls appliance) PROTOCOL (two-way communication of specification & state) PROTOCOL (two-way communication of specification & state) device specification & state feedback COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) COMMUNICATION (802.11, Bluetooth, Zigbee, etc.) control

19. Controlling Appliances We have built adaptors for many actual appliances • Sony Digital Camcorder • Windows Media Player • Axis UPnP Pan & Tilt Camera • Lutron Lighting • X10 Lighting • Audiophase Shelf Stereo • AudioReQuest MP3 player • GM Vehicle Information System • GM Vehicle Climate Control Written specifications for others • Elevator • Telephone/Answering Machine • GM Navigation System • Several Alarm Clocks

20. Outline • Introduction • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Conclusion

21. High-Level Conventions Problem • Human designers rely partly on conventions when making an interface • Users expect their appliances to use conventions they know about • How do we integrate high-level info into PUC system?

22. Smart Templates Method for specifying high-level information to interface generators Solution • Mark groups with tags that identify high-level information media-controls, phone-dialpad, time, date, etc. • Restrict the contents of groups so that interface generators can interpret the high-level meaning • Standardize the tags and restrictions in advance, so that designers know what interface generators expect

23. Smart Templates, cont. Features • Parameterized • Specified using primitive elements of specification language Renderable by any interface generator • Generators can contain special code that renders a template as an interface designer might

24. Preliminary Implementation A few templates: image, image-list, media-controls, time-duration Interfaces with Smart Templates

25. Continuing Work Define and implement Smart Templates • date, mute, power, time-absolute, volume, etc. • Use to improve renderings of list interfaces • Develop more as more appliances are specified Combinations of templates • Less implementation cost than a new template • e.g. date and time-absolute Use of templates with data already on controller device • e.g. calendar and address information • Might allow user to enter address from contact list into navigation system

26. Outline • Introduction • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Conclusion

27. Interface Consistency PUC devices have a unique opportunity to provide consistency for the user • Personal device • Used for interacting with most appliances Two ways that PUC UIs can be made consistent • With other applications on the same device • With past interfaces for similar appliances

28. Consistency with Past Interfaces Two sub-problems to address: • Similarity Which functions of a new appliance are similar to the functions of interfaces generated in the past? • Consistency For similar functions, what rules from previous interfaces can be applied to ensure consistency?

29. Similarity Problem Difficult to conclusively know whether two functions from different appliances are the same • Very little semantic information in the specification language Can estimate similarity based on properties of state variables • Smart Template • Name • Group Name • Labels • Type Improve estimate by looking at relationships between multiple similar variables

30. Similarity & Consistency Problems new previous sparse similarity new previous new previous branch similarity significant similarity

31. Outline • Introduction • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Conclusion

32. The “Experience”

33. The “Experience”, cont. (MIT dorm) How can a PUC device provide improved interfaces for connected systems?

34. Improving Interfaces with PUC Generate an interface that aggregates all functions into one set of screens Organized by task instead of appliance Automatically create macros for frequently used functions e.g. “Play DVD” would: • turn on television, DVD player, stereo • turn off VCR • set the TV and stereo sources to the DVD player • instruct the user to insert a DVD (if necessary) • play the DVD How can appliance descriptions support these features?

35. Model Data Flow Define inputs and outputs of each appliance • Including definition of whether output is human consumable (e.g. screen, speakers, etc.) • May need to define which appliances are content sources and sinks Develop a basic language for describing how states and commands modify I/O • Accept/Reject input i • Activate/Modify/Deactivate output o

36. Distributed Task Language Data flow helps us find cross-appliance functions, but does not tell us much about likelihood of use • Need to be able to filter Add some task information • Store task information within each appliance description Use an existing task language • Combine sub-tasks from each appliance in a connected system to create complete tasks

37. Outline • Introduction • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Conclusion

38. Two Goals for PUC System Breadth The appliance specification language is capable of describing a wide variety of appliances Quality Interfaces generated for specifications across that range beat the usability of the manufacturers’ interfaces for the same appliances How do I validate that these goals are met?

39. Evaluation Evaluation Method • Develop a list of appliances that are interesting • Complexity • Unique feature • Representative of a class of appliances • Specify each appliance • Perform a comparative user study on several appliances to test quality

40. Evaluation, cont. Interface consistency • Train users to be experts with one interface and test performance on a related appliance interface Control for consistency features • Test the variability in appliance specifications Multi-appliance UIs • Compare performance of users with/without multi-appliance features Focus on structure, selection of multi-appliance functions

41. Outline • Introduction • Personal Universal Controller (PUC) System • Handling High-Level Conventions • Interface Consistency • Generating Interfaces for the “Experience” • Validation • Conclusion

42. Conclusion Problem • Appliances are increasingly complex • Appliance user interfaces are often hard to use Solution • Move appliance interfaces to a mobile device the user is already carrying • Automatically generate interfaces so that: Interfaces are customized for the device and modality that the user prefers Interfaces for similar appliances can be made consistent Interfaces for multiple appliances can be combined into a single interface • Validate generated interfaces to prove high quality claim

43. Thesis Committee Brad A. Myers (chair) Scott Hudson John Zimmerman Dan Olsen Jr. Funding National Science Foundation Microsoft General Motors Intel Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) VividLogic Lucent Lutron Lantronix Nokia PUC Project Members Kevin Litwack Thomas K. Harris Michael Higgins Joseph Hughes Roni Rosenfeld Rajesh Seenichamy Pegeen Shen Htet Htet Aung Mathilde Pignol Suporn Pongnumkul Stefanie Shriver Jeffrey Stylos Peter Lucas Thomas Psik Acknowledgements Collaborators & Friends • Naomi Ramos • Desney Tan • Daniel Avrahami • Gaetano Borriello • Laura Dabbish • Andrew Faulring • James Fogarty • Krzysztof Gajos • Darren Gergle • Andy Ko • Amy Nichols • Mick Nichols • Sally Nichols • Trevor Pering • Fleming Seay • Irina Shklovski • Roy Want • Jake Wobbrock • and many others…

44. Thanks for listening! For more information… http://www.cs.cmu.edu/~pebbles/puc/ http://www.cs.cmu.edu/~jeffreyn/

45. Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Symposium Presentation Seventeenth Annual ACM Symposium on User Interface Software and Technology October 24, 2004

49. XML-based Specification Language Describes appliance with these features: • Functions of Device State Variables and Commands • Labeling Multiple labels are necessary • Grouping Hierarchical groups • Dependency Information For enabling and structure

50. Outside the Scope • Help systems for generated interfaces • Automated trouble-shooting for complex systems • Service Discovery • Macros and End-User Programming • Security • Inter-operability with INCITS/V2