Automatically generating high quality user interfaces for appliances
This presentation is the property of its rightful owner.
Sponsored Links
1 / 75

Automatically Generating High-Quality User Interfaces for Appliances PowerPoint PPT Presentation


  • 100 Views
  • Uploaded on
  • Presentation posted in: General

Automatically Generating High-Quality User Interfaces for Appliances. Jeffrey Nichols Thesis Proposal Presentation Thesis Committee: Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr. Appliances are complex and their user interfaces are often hard to use!. Problem, cont.

Download Presentation

Automatically Generating High-Quality User Interfaces for Appliances

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Automatically generating high quality user interfaces for appliances

Automatically Generating High-Quality User Interfaces for Appliances

Jeffrey Nichols

Thesis Proposal Presentation

Thesis Committee:

Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr.


Problem cont

Appliances are complex and their user interfaces are often hard to use!

Problem, cont.


Problem cont1

Problem, cont.


Problem cont2

Problem, cont.

May 2003

CNN – Jan 2004

April 29, 1991


Thesis statement

Thesis Statement

“A system can automatically generate user interfaces on a wide variety of platforms for remotely controlling appliances where the user’s performance is better than with the manufacturers’ interfaces for the appliances.”


Approach use mobile devices

Approach: Use Mobile Devices

Common, cheap, and often have ability to communicate

10 million smart phones sold 11.2 million handheld units shipped in 2003120 million mobile phone subscribers


Approach cont

Approach, cont.

Specifications

Control

Feedback

Use mobile devices to control all appliances in the environment

Mobile Devices

Appliances

Key Features

Two-way communication, Abstract Descriptions, Multiple Platforms, Automatic Interface Generation


Automatic generation of uis

Automatic Generation of UIs

Benefits

  • Multiple modalities (GUI + Speech UI)

  • All interfaces consistent for a user

    • With conventions of the handheld

    • Even from multiple manufacturers

      Addresses hotel alarm clock problem

  • Create user interfaces that control multiple connected appliances

    e.g. a home theater


Important work by others

Important Work By Others

INCITS/V2 Standardization Effort

Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002]

Controlling Appliances

  • Xweb [Olsen Jr., UIST 2000]

  • Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001]

  • Speakeasy [Newman, UIST 2002]

    Model-based User Interface Work

  • UIDE [Sukaviriya, CHI 1993], HUMANOID [Szekely, CHI 1993]

  • MASTERMIND [Szekely 1995], TRIDENT [Vanderdonckt 1995]

  • eXtensible Interface Markup Language (XIML) [Puerta, IUI 2002]

  • User Interface Modeling Language (UIML) [Abrams, W3C 1999]

    None of these systems have been validated to generate high-quality interfaces without designer intervention.


Research approach

Research Approach

  • Hand-design remote control interfaces

  • Determine functional information needed from appliances to design user interfaces

  • Design language for describing appliance functions

  • Build interface generators for multiple platforms

  • Perform user studies to evaluate the interface generators


Outline

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


Initial approach

Initial Approach

What information is needed about the appliance to automatically generate remote control interfaces?

Investigate via a design process

Create interfaces by hand

  • AIWA Shelf Stereo

  • AT&T Telephone/Answering Machine

    Improve quality with heuristic analysis and think-aloud studies with several users

    Compare interfaces with actual appliance interfaces to validate PUC concept

    Analyze interfaces for functional information


Hand designed interfaces

Hand-Designed Interfaces

Interfaces for Microsoft’s PocketPC (simulated remote control)

telephone

stereo


Comparison study

Comparison Study

Compared performance of first-time users (not experts)

Procedure

  • Each subject worked two sets of tasks

    both stereo and phone

    controlled for order and interface

    Performance Metrics

  • Time to complete all tasks

  • Number of times a user manual was needed

  • Number of missteps


Comparison study results

Comparison Study Results

Using our interfaces, users were twice as fastand made half as many errors

All differences are significant (p < 0.05)


Outline1

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


Architecture

Architecture

APPLIANCES

(Stereo, Alarm Clock, etc.)

PUC DEVICES

(automatic interface generation)

ADAPTOR

(publishes description +

appliance state + controls appliance)

PROTOCOL

(two-way communication

of specification & state)

PROTOCOL

(two-way communication

of specification & state)

device specification & state feedback

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

control


Architecture1

Architecture

APPLIANCES

(Stereo, Alarm Clock, etc.)

PUC DEVICES

(automatic interface generation)

ADAPTOR

(publishes description +

appliance state + controls appliance)

PROTOCOL

(two-way communication

of specification & state)

PROTOCOL

(two-way communication

of specification & state)

device specification & state feedback

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

control


Architecture2

Architecture

APPLIANCES

(Stereo, Alarm Clock, etc.)

PUC DEVICES

(automatic interface generation)

ADAPTOR

(publishes description +

appliance state + controls appliance)

PROTOCOL

(two-way communication

of specification & state)

PROTOCOL

(two-way communication

of specification & state)

device specification & state feedback

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

control


Architecture3

Architecture

APPLIANCES

(Stereo, Alarm Clock, etc.)

PUC DEVICES

(automatic interface generation)

ADAPTOR

(publishes description +

appliance state + controls appliance)

PROTOCOL

(two-way communication

of specification & state)

PROTOCOL

(two-way communication

of specification & state)

device specification & state feedback

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

control


Architecture4

Architecture

APPLIANCES

(Stereo, Alarm Clock, etc.)

PUC DEVICES

(automatic interface generation)

ADAPTOR

(publishes description +

appliance state + controls appliance)

PROTOCOL

(two-way communication

of specification & state)

PROTOCOL

(two-way communication

of specification & state)

device specification & state feedback

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

COMMUNICATION

(802.11, Bluetooth, Zigbee, etc.)

control


Language design

Language Design

Informed by hand-designed interfaces

  • What functional information was needed to create interfaces?

    Additional Requirements

  • Support complete functionality of appliance

  • No specific layout information

  • Only one way to specify anything

Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/


Language elements

Language Elements

Elements

  • State variables & commands

  • Labels

  • Group tree

  • Dependency information

    Example media player specification

  • Play, stop, pause, next track, previous track

  • Play list


Language elements cont

Language Elements, cont.

State Variables and Commands

  • Represent functions of appliance

  • State variables have types

    • Boolean, Enumeration, Integer, String, etc.

  • Variables sufficient for most functions but not all

    • e.g. “seek” button on a Radio


Language elements cont1

Language Elements, cont.

Label Information

One label not suitable everywhere

  • The optimal label length changes with screen size

  • Speech interfaces may benefit from pronunciation and text-to-speech information

    “Label Dictionary”

  • A group of semantically similar labels

  • Different lengths

  • Information for different modalities


Language elements cont2

Language Elements, cont.

Label Information

One label not suitable everywhere

  • The optimal label length changes with screen size

  • Speech interfaces may benefit from pronunciation and text-to-speech information

    “Label Dictionary”

  • A group of semantically similar labels

  • Different lengths

  • Information for different modalities


Language elements cont3

Language Elements, cont.

Group Tree

  • Specify organization of functions

  • We use n-ary tree with variables or commands at leaves

  • Also used for specifying complex types

    Lists

    Unions


Language elements cont4

Language Elements, cont.

Group Tree

  • Specify organization of functions

  • We use n-ary tree with variables or commands at leaves

  • Also used for specifying complex types

    Lists

    Unions


Language elements cont5

Language Elements, cont.

Dependency Information

  • Formulas that specify when a variable or command is active in terms of other state variables

    Equals, Greater Than, Less Than, Is Defined

    Linked with logical operators (AND, OR)

  • Allows feedback to user when a function is not available


Interface generators

Interface Generators

Generators for Two Modalities

Graphical

  • Desktop, PocketPC, and Microsoft Smartphone

    Speech

  • Ph.D. work of Thomas Harris

  • Built on top of the PUC framework

  • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]


Graphical interface generator

Graphical Interface Generator

Rule-based approach

  • Multiple phases that iteratively transform a specification into a user interface

    Focuses on panel structure of user interface

  • Small groups of controls have basic layouts

  • Complexity comes from structure of groups

  • Structure can be inferred from dependency info!


Generation process

Generation Process

  • Determine conceptual layout

    • Infer panel structure from dependencies using “mutual exclusion” property

    • Choose controls (decision tree)

    • Choose row layout (one column, two column, etc.)

  • Allocate space

    • Examine panel contents and choose sizes

  • Instantiate and place controls

  • Fix layout problems


Generation process1

Generation Process

  • Determine conceptual layout

    • Infer panel structure from dependencies using “mutual exclusion” property

    • Choose controls (decision tree)

    • Choose row layout (one column, two column, etc.)

  • Allocate space

    • Examine panel contents and choose sizes

  • Instantiate and place controls

  • Fix layout problems


Generation process2

Generation Process

  • Determine conceptual layout

    • Infer panel structure from dependencies using “mutual exclusion” property

    • Choose controls (decision tree)

    • Choose row layout (one column, two column, etc.)

  • Allocate space

    • Examine panel contents and choose sizes

  • Instantiate and place controls

  • Fix layout problems


Generation process3

Generation Process

  • Determine conceptual layout

    • Infer panel structure from dependencies using “mutual exclusion” property

    • Choose controls (decision tree)

    • Choose row layout (one column, two column, etc.)

  • Allocate space

    • Examine panel contents and choose sizes

  • Instantiate and place controls

  • Fix layout problems


Generation process4

Generation Process

  • Determine conceptual layout

    • Infer panel structure from dependencies using “mutual exclusion” property

    • Choose controls (decision tree)

    • Choose row layout (one column, two column, etc.)

  • Allocate space

    • Examine panel contents and choose sizes

  • Instantiate and place controls

  • Fix layout problems


Generation process5

Generation Process

With layout fixing rules

Without layout fixing rules

  • Determine conceptual layout

    • Infer panel structure from dependencies using “mutual exclusion” property

    • Choose controls (decision tree)

    • Choose row layout (one column, two column, etc.)

  • Allocate space

    • Examine panel contents and choose sizes

  • Instantiate and place controls

  • Fix layout problems


Outline2

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


High level conventions

High-Level Conventions

Problem

  • Human designers rely partly on conventions when making an interface

  • Users expect their appliances to use conventions they know about


Smart templates

Smart Templates

Need a way to specify high-level information to interface generators

Solution

  • Mark groups with tags that identify high-level information

    media-controls, phone-dialpad, time, date, etc.

  • Restrict the contents of groups so that interface generators can interpret the high-level meaning

  • Standardize the tags and restrictions in advance, so that designers know what interface generators expect


Smart templates cont

Smart Templates, cont.

Features

  • Parameterized

  • Specified using primitive elements of specification language


Interfaces with smart templates

Preliminary Implementation

A few templates: image, image-list, media-controls, time-duration

Interfaces with Smart Templates


Continuing work

Continuing Work

Define and implement Smart Templates

  • date, mute, power, time-absolute, volume, etc.

  • Develop more as more appliances are specified

    Combinations of templates

  • Less implementation cost than a new template

  • e.g. date and time-absolute

    Use of templates with data already on controller device

  • e.g. calendar and address information

  • Might allow user to enter address from contact list into navigation system


Outline3

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


Interface consistency

Interface Consistency

PUC devices have a unique opportunity to provide consistency for the user

  • Personal device

  • Used for interacting with most appliances

    Two ways that PUC UIs can be made consistent

  • With other applications on the same device

  • With past interfaces for similar appliances


Consistency with past interfaces

Consistency with Past Interfaces

Two sub-problems to address:

  • Similarity

    Which functions of a new appliance are similar to the functions of interfaces generated in the past?

  • Consistency

    For similar functions, what rules from previous interfaces can be applied to ensure consistency?


Similarity problem

Similarity Problem

Difficult to conclusively know whether two functions from different appliances are the same

  • Very little semantic information in the specification language

    Can estimate similarity based on properties of state variables

  • Smart Template

  • Name

  • Group Name

  • Labels

  • Type

    Improve estimate by looking at relationships between multiple similar variables


Similarity consistency problems

Similarity & Consistency Problems

new

previous

sparse similarity

new

previous

new

previous

branch similarity

significant similarity


Outline4

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


The experience

The “Experience”


The experience cont mit dorm

The “Experience”, cont. (MIT dorm)

How can a PUC device provide improved interfaces for connected systems?


Improving interfaces with puc

Improving Interfaces with PUC

Generate an interface that aggregates all functions into one set of screens

Organized by task instead of appliance

Automatically create macros for frequently used functions

e.g. “Play DVD” would:

  • turn on television, DVD player, stereo

  • turn off VCR

  • set the TV and stereo sources to the DVD player

  • instruct the user to insert a DVD (if necessary)

  • play the DVD

    How can appliance descriptions support these features?


Distributed task language

Distributed Task Language

My plan:

  • Store task information within each appliance description

    Use an existing task language

  • Combine sub-tasks from each appliance in a connected system to create complete tasks


Outline5

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


Two goals for puc system

Two Goals for PUC System

Breadth

The appliance specification language is capable of describing a wide variety of appliances

Quality

Interfaces generated for specifications across that range beat the usability of the manufacturers’ interfaces for the same appliances

How do I validate that these goals are met?


Evaluation

Evaluation

Evaluation Method

  • Develop a list of appliances that are interesting

    • Complexity

    • Unique feature

    • Representative of a class of appliances

  • Specify each appliance

  • Perform a comparative user study on several appliances to test quality

    Also need to explore expert usage of interfaces…


Appliance list so far comments

Appliance List (so far…comments?)


Outline6

Outline

  • Introduction

  • Hand-Designed Interfaces and User Studies

  • Personal Universal Controller (PUC) System

  • Handling High-Level Conventions

  • Interface Consistency

  • Generating Interfaces for the “Experience”

  • Validation

  • Proposed Contributions & Schedule


Proposed contributions

Proposed Contributions

  • An abstract appliance modeling language for describing the functionality of appliances

  • Algorithms for automatically generating high quality interfaces from that language

  • The Smart Templates technique for incorporating high-level conventions

  • Algorithms for ensuring consistency across generated interfaces

    and…


Proposed contributions cont

Proposed Contributions, cont.

  • A distributedtask-modeling language for describing the sub-tasks specific to an appliance in a multi-appliance system

  • Algorithms for automatically generating a single user interface for multiple appliances using distributed task information

  • Interface generation software on multiple platforms that is shown by user testing to be better than manufacturers’ interfaces for the same appliances


Proposed schedule

Proposed Schedule

1

2

3

4

5

6

7

8

March 2004

June

September

December

March2005

June

September

December2005

  • Interface consistency

  • Smart Templates

  • Preliminary user studies with GM vehicle interfaces

  • Improve robustness and quality of interface generators on all platforms

  • Develop distributed task modeling language and multi-appliance user interface generation

  • Implement appliances for final user study

  • Conduct final user study and iterate UI generation rules, if necessary

  • Write dissertation


Conclusion

Conclusion

Problem

  • Appliances are increasingly complex

  • Appliance user interfaces are often hard to use

    Solution

  • Move appliance interfaces to a mobile device the user is already carrying

  • Automatically generate interfaces so that:

    Interfaces are customized for the device and modality that the user prefers

    Interfaces for similar appliances can be made consistent

    Interfaces for multiple appliances can be combined into a single interface

  • Validate generated interfaces to prove high quality claim


Acknowledgements

Thesis Committee

Brad A. Myers (chair)

Scott Hudson

John Zimmerman

Dan Olsen Jr.

Funding

National Science Foundation

Microsoft

General Motors

Intel

Pittsburgh Digital Greenhouse

Equipment Grants

Mitsubishi (MERL)

VividLogic

Lucent

Lutron

Lantronix

Nokia

PUC Project Members

Kevin Litwack

Thomas K. Harris

Michael Higgins

Joseph Hughes

Roni Rosenfeld

Rajesh Seenichamy

Pegeen Shen

Htet Htet Aung

Mathilde Pignol

Suporn Pongnumkul

Stefanie Shriver

Jeffrey Stylos

Peter Lucas

Acknowledgements

Collaborators & Friends

  • Naomi Ramos

  • Desney Tan

  • Daniel Avrahami

  • Gaetano Borriello

  • Laura Dabbish

  • Andrew Faulring

  • James Fogarty

  • Krzysztof Gajos

  • Darren Gergle

  • Andy Ko

  • Amy Nichols

  • Mick Nichols

  • Sally Nichols

  • Trevor Pering

  • Fleming Seay

  • Irina Shklovski

  • Roy Want

  • Jake Wobbrock

  • and many others…


Thanks for listening

Thanks for listening!

For more information…

http://www.cs.cmu.edu/~pebbles/puc/

http://www.cs.cmu.edu/~jeffreyn/


Automatically generating high quality user interfaces for appliances1

Automatically Generating High-Quality User Interfaces for Appliances

Jeffrey Nichols

Thesis Proposal Presentation

Thesis Committee:

Brad A. Myers (chair), Scott E. Hudson, John Zimmerman, Dan Olsen Jr.


Outside the scope

Outside the Scope

  • Help systems for generated interfaces

  • Automated trouble-shooting for complex systems

  • Service Discovery

  • Macros and End-User Programming

  • Security

  • Inter-operability with INCITS/V2


Palm interfaces

Palm Interfaces

Initially designed paper-prototype interfaces for Palm

telephone

stereo


Extra group tree

Extra Group Tree


Choosing panel structure

Choosing Panel Structure

a)

b)

c)

partial screen

full screen

tabbed


Controlling appliances

Controlling Appliances

We have built adaptors for many actual appliances

  • Sony Digital Camcorder

  • Windows Media Player

  • Axis UPnP Pan & Tilt Camera

  • Lutron Lighting

  • X10 Lighting

  • Audiophase Shelf Stereo

  • AudioReQuest MP3 player

  • GM Vehicle Information System

  • GM Vehicle Climate Control

    Written specifications for others

  • Elevator

  • Telephone/Answering Machine

  • GM Navigation System

  • Several Alarm Clocks


Smart templates example

Smart Templates Example


Smart templates cont1

Smart Templates, cont.

One template can be used across multiple appliances and can be generated on multiple platforms


Consistency problem

Consistency Problem

Apply different consistency techniques depending on similarity

  • Sparse similarity

    Use similar controls for similar functions

    e.g. use a scroll bar for new volume, even if this would not be the normal representation (might be a combo box)

  • Branch similarity

    Use the previous interface structure of branch in the new interface

    e.g. branch of previous interface was shown in a tabbed panel. In the new interface, separate functions of new interface with tabs and have similar functions in a tabbed panel.

  • Significant similarity

    Use the group tree from the previously generated interface

    Interpolate extra functions into the tree


The experience cont

The “Experience”, cont.

How can a PUC device provide improved interfaces for these systems?

Users often think of multiple connected appliances as one system

  • Home theater

  • CMU lecture room system

  • others…


  • Login