Generating remote control interfaces for complex appliances
1 / 27

Generating Remote Control Interfaces for Complex Appliances - PowerPoint PPT Presentation

  • Uploaded on

Generating Remote Control Interfaces for Complex Appliances. Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October 30, 2002. The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France. The Problem. Appliances are too complex.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Generating Remote Control Interfaces for Complex Appliances' - doane

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Generating remote control interfaces for complex appliances

Generating Remote Control Interfaces for Complex Appliances

Jeffrey Nichols and Brad A. Myers

Carnegie Mellon University

October 30, 2002

The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France

The problem
The Problem

Appliances are too complex

The problem cont
The Problem, cont.

Each complex appliances has its own idiosyncratic interface!

  • Stereo systems

  • Telephones

  • VCRs

  • Alarm Clocks

  • Increasingly Computerized

    Low Usability

Our solution




Our Solution

Separate the interface from the appliance!

Handheld becomes personal universal controller (PUC)

Key Features

  • Interface-independent appliance specification

  • Automatic generation of GUI and speech interfaces

Automatic generation of uis
Automatic Generation of UIs


  • All interfaces consistent for the user

    • With conventions of handheld

    • Even from multiple manufacturers

      Addresses hotel alarm clock problem!

  • Multiple modalities (GUI + Speech UI)

  • Can take into account user preferences

  • Will work on special purpose devices (for disabled)

Important work by others
Important Work By Others

  • INCITS V2 Standardization Effort

    Alternative Interface Access Protocol (AIAP)

    [Zimmermann, CHI 2002]

  • User Interface Modeling Language (UIML)


  • Xweb (now ICE)

    [Olsen Jr., UIST 2000]

  • Stanford iRoom, iCrafter

    [Ponnekanti, Ubicomp 2001]

  • Speakeasy

    [Newman, UIST 2002]


- Comm. Protocol

- Interface Generators

- Specification Lang.

- Appliance Adaptors


Language design
Language Design


  • Create reference interfaces

    • AIWA Shelf Stereo

    • AT&T Telephone/Answering Machine

  • Test interfaces with subjects

    • Users twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfaces

  • Analyze interfaces for functional information

Language elements
Language Elements

State Variables and Commands

  • Represent functions of appliance

  • State variables have types

    • Boolean, Enumeration, Integer, String, etc.

  • Variables sufficient for most functions but not all

    • “seek” button on a Radio

      Label Information

  • One label not suitable everywhere

    • The optimal label length changes with screen size

    • Speech interfaces may benefit from pronunciation and text-to-speech information

Language elements cont
Language Elements, cont.

Group Tree

  • Specify organization of functions

  • We use n-ary tree with variables or commands at leaves

Language elements cont1
Language Elements, cont.

Dependency Information

  • Formulas that specify when a variable or command is active in terms of other state variables

    • Equals, Greater Than, Less Than

    • Linked with logical operators (AND, OR)

  • For example,

    <and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>

Interface generators
Interface Generators

Generators for Two Modalities

  • Graphical

    • Implemented for PocketPC in Java 1.1

    • Uses dependency information to generate panel structure of interface

  • Speech

    • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

    • Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)

Graphical interface generator
Graphical Interface Generator

Focuses on panel structure of user interface

  • Small groups of controls have basic layouts

  • Complexity comes from structure of groups

  • Structure can be inferred from dependency info!

Inferring structure
Inferring Structure

Find sets of variables that are “mutually exclusive”

  • Every variable in a set will never be active at the same time as a variable in another set

    Create structure with sets, using overlapping panels

Choosing panel types
Choosing Panel Types




partial screen

full screen


Making the interface concrete
Making the Interface Concrete

Finish conceptual layout

  • Choose controls (decision tree)

  • Choose row layouts

    (one column, two column, etc.)

    Allocate space

  • Examine panel contents and choose sizes

    Instantiate and place controls

Generating speech interfaces
Generating Speech Interfaces

Automatically build USI tree from dependencies

  • Allows verbal navigation of functional groups

    Automatically generate grammar for parser

  • Phrases for query and control

    “What is playmode?”

    “Set playmode to play”


    Automatically generate language model and pronunciation for recognizer

Controlling appliances
Controlling Appliances

We have built adaptors for many actual appliances

  • Sony Digital Camcorder

  • X10 Lighting

  • Audiophase Shelf Stereo

  • AudioReQuest MP3 player

  • Software Media Players (WinAmp, WMP)

    Written specifications for others

  • Elevator

  • Telephone/Answering Machine

Future work
Future Work

  • Improve quality of generated user interfaces

    • Back-tracking, etc.

  • Specification and generation of list interfaces

    • Necessary for many appliances

  • Adaptive interface generation

    • New interfaces modeled on past interfaces

  • General purpose appliance adaptor for HAVi and UPnP protocols


PUC Project Members

Brad A. Myers

Michael Higgins

Joseph Hughes

Thomas K. Harris

Roni Rosenfeld

Mathilde Pignol

Stefanie Shriver

Peter Lucas

Kevin Litwack


National Science Foundation

Pittsburgh Digital Greenhouse


Equipment Grants

Mitsubishi (MERL)


Symbol Technologies






User Interface Software & Technology (UIST)

October 30, 2002

Determining panel structure
Determining Panel Structure

  • Find states that others depend upon

  • Search for the group that contains this state

  • Determine mutually exclusive sets within group children

  • Assign sets to panels

    • Based upon types, tree properties


Mode == Any








Mode == Radio




Mode == CD

Choosing panel types1
Choosing Panel Types

Rules for Choosing Panels

  • If…

    • state has boolean type

    • all controls are one set

  • Create…

    • Two full-screen overlapping panels.

Choosing panel types cont
Choosing Panel Types, cont.

Rules for Choosing Panels

  • If…

    • state has enumerated type

    • there is a set for every value of the state

  • Create…

    • a tabbed panel

Choosing panel types cont1
Choosing Panel Types, cont.

Rules for Choosing Panels

  • If…

    • there are mutually exclusive sets of controls

  • Create…

    • overlapping panels

    • a control for the state not on the overlapping panel