Generating Remote Control Interfaces for Complex Appliances - PowerPoint PPT Presentation

Generating remote control interfaces for complex appliances
1 / 27

  • Uploaded on
  • Presentation posted in: General

Generating Remote Control Interfaces for Complex Appliances. Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October 30, 2002. The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France. The Problem. Appliances are too complex.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Generating Remote Control Interfaces for Complex Appliances

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Generating remote control interfaces for complex appliances

Generating Remote Control Interfaces for Complex Appliances

Jeffrey Nichols and Brad A. Myers

Carnegie Mellon University

October 30, 2002

The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France

The problem

The Problem

Appliances are too complex

The problem cont

The Problem, cont.

Each complex appliances has its own idiosyncratic interface!

  • Stereo systems

  • Telephones

  • VCRs

  • Alarm Clocks

  • Increasingly Computerized

    Low Usability

Our solution




Our Solution

Separate the interface from the appliance!

Handheld becomes personal universal controller (PUC)

Key Features

  • Interface-independent appliance specification

  • Automatic generation of GUI and speech interfaces

Automatic generation of uis

Automatic Generation of UIs


  • All interfaces consistent for the user

    • With conventions of handheld

    • Even from multiple manufacturers

      Addresses hotel alarm clock problem!

  • Multiple modalities (GUI + Speech UI)

  • Can take into account user preferences

  • Will work on special purpose devices (for disabled)

Important work by others

Important Work By Others

  • INCITS V2 Standardization Effort

    Alternative Interface Access Protocol (AIAP)

    [Zimmermann, CHI 2002]

  • User Interface Modeling Language (UIML)


  • Xweb (now ICE)

    [Olsen Jr., UIST 2000]

  • Stanford iRoom, iCrafter

    [Ponnekanti, Ubicomp 2001]

  • Speakeasy

    [Newman, UIST 2002]



- Comm. Protocol

- Interface Generators

- Specification Lang.

- Appliance Adaptors


Language design

Language Design


  • Create reference interfaces

    • AIWA Shelf Stereo

    • AT&T Telephone/Answering Machine

  • Test interfaces with subjects

    • Users twice as fast and made half the errors with reference interfaces as compared to manufacturers’ interfaces

  • Analyze interfaces for functional information

Language elements

Language Elements

State Variables and Commands

  • Represent functions of appliance

  • State variables have types

    • Boolean, Enumeration, Integer, String, etc.

  • Variables sufficient for most functions but not all

    • “seek” button on a Radio

      Label Information

  • One label not suitable everywhere

    • The optimal label length changes with screen size

    • Speech interfaces may benefit from pronunciation and text-to-speech information

Language elements cont

Language Elements, cont.

Group Tree

  • Specify organization of functions

  • We use n-ary tree with variables or commands at leaves

Language elements cont1

Language Elements, cont.

Dependency Information

  • Formulas that specify when a variable or command is active in terms of other state variables

    • Equals, Greater Than, Less Than

    • Linked with logical operators (AND, OR)

  • For example,

    <and> <equals state=“PowerState”>true</equals> <equals state=“RadioBand”>AM</equals></and>

Interface generators

Interface Generators

Generators for Two Modalities

  • Graphical

    • Implemented for PocketPC in Java 1.1

    • Uses dependency information to generate panel structure of interface

  • Speech

    • Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001]

    • Uses dependency information to disambiguate shortcut words (e.g. “play”) and resolve pre-conditions for a requested function (e.g. “play CD”)

Graphical interface generator

Graphical Interface Generator

Focuses on panel structure of user interface

  • Small groups of controls have basic layouts

  • Complexity comes from structure of groups

  • Structure can be inferred from dependency info!

Inferring structure

Inferring Structure

Find sets of variables that are “mutually exclusive”

  • Every variable in a set will never be active at the same time as a variable in another set

    Create structure with sets, using overlapping panels

Choosing panel types

Choosing Panel Types




partial screen

full screen


Making the interface concrete

Making the Interface Concrete

Finish conceptual layout

  • Choose controls (decision tree)

  • Choose row layouts

    (one column, two column, etc.)

    Allocate space

  • Examine panel contents and choose sizes

    Instantiate and place controls

Generating speech interfaces

Generating Speech Interfaces

Automatically build USI tree from dependencies

  • Allows verbal navigation of functional groups

    Automatically generate grammar for parser

  • Phrases for query and control

    “What is playmode?”

    “Set playmode to play”


    Automatically generate language model and pronunciation for recognizer

Controlling appliances

Controlling Appliances

We have built adaptors for many actual appliances

  • Sony Digital Camcorder

  • X10 Lighting

  • Audiophase Shelf Stereo

  • AudioReQuest MP3 player

  • Software Media Players (WinAmp, WMP)

    Written specifications for others

  • Elevator

  • Telephone/Answering Machine



Future work

Future Work

  • Improve quality of generated user interfaces

    • Back-tracking, etc.

  • Specification and generation of list interfaces

    • Necessary for many appliances

  • Adaptive interface generation

    • New interfaces modeled on past interfaces

  • General purpose appliance adaptor for HAVi and UPnP protocols


PUC Project Members

Brad A. Myers

Michael Higgins

Joseph Hughes

Thomas K. Harris

Roni Rosenfeld

Mathilde Pignol

Stefanie Shriver

Peter Lucas

Kevin Litwack


National Science Foundation

Pittsburgh Digital Greenhouse


Equipment Grants

Mitsubishi (MERL)


Symbol Technologies






User Interface Software & Technology (UIST)

October 30, 2002

Determining panel structure

Determining Panel Structure

  • Find states that others depend upon

  • Search for the group that contains this state

  • Determine mutually exclusive sets within group children

  • Assign sets to panels

    • Based upon types, tree properties


Mode == Any








Mode == Radio




Mode == CD

Choosing panel types1

Choosing Panel Types

Rules for Choosing Panels

  • If…

    • state has boolean type

    • all controls are one set

  • Create…

    • Two full-screen overlapping panels.

Choosing panel types cont

Choosing Panel Types, cont.

Rules for Choosing Panels

  • If…

    • state has enumerated type

    • there is a set for every value of the state

  • Create…

    • a tabbed panel

Choosing panel types cont1

Choosing Panel Types, cont.

Rules for Choosing Panels

  • If…

    • there are mutually exclusive sets of controls

  • Create…

    • overlapping panels

    • a control for the state not on the overlapping panel

  • Login