mobile devices for control of ubiquitous multimedia l.
Skip this Video
Loading SlideShow in 5 Seconds..
Mobile Devices for Control of Ubiquitous Multimedia PowerPoint Presentation
Download Presentation
Mobile Devices for Control of Ubiquitous Multimedia

Loading in 2 Seconds...

play fullscreen
1 / 47

Mobile Devices for Control of Ubiquitous Multimedia - PowerPoint PPT Presentation

  • Uploaded on

Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Mobile Devices for Control of Ubiquitous Multimedia Brad A. Myers Handheld Devices Typically, handheld devices for: Communication

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

Mobile Devices for Control of Ubiquitous Multimedia

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
mobile devices for control of ubiquitous multimedia

Human Computer Interaction Institute

School of Computer Science

Carnegie Mellon University

Mobile Devices for Control ofUbiquitous Multimedia

Brad A.

handheld devices
Handheld Devices
  • Typically, handheld devices for:
    • Communication
    • As a “Personal Information Manager”
    • As a “Personal Digital Assistant”
    • As a PC replacement
      • “PocketPC”
    • As a media player
  • Handheld Devices for Remote Control of other devices and media
    • Remote Control of PCs
    • Remote Control of appliances
  • Appliances are too complex
  • Too many remotes
  • April 29, 1991
  • Use a personal handheld device as an Interface to the PC and to the Appliances
history of communication
History of Communication
  • ParcTab – continuous communication
  • Early “organizers” – no communication
  • Newton – extra cost for synchronization
  • Palm – HotSync once a day
  • WindowsCE/PocketPC – ActiveSync once a day
now handhelds are communicating
Now, Handhelds are Communicating

Toshiba e740

  • 802.11
  • BlueTooth
  • Cell-phone network (G3)

HP iPaq 3870

Microsoft SmartPhone

pebbles project
Pebbles Project
  • Performed as part of the Pebbles Project
  • Overall goal: Use of multiple devicesat the same time
    • Palm and desktop PC
    • Mobile Phone and “Smart Home”
    • Multiple handhelds in a meeting
    • Pocket PC and appliances
  • Multiple users with their devices
  • Single user with multiple devices
pebbles is
Pebbles is:








DAs for

ntry of


ytes and

ocations from



pebbles for multimedia
Pebbles for MultiMedia
  • How will handheld devices fit into the user’s overall media space?
    • Where does media come from?
    • Where is the media is displayed?
    • How is the media controlled?
  • How can handheld devices improve the user interfaces of other media devices, rather than just being another gadget to master?
controlling a pc

Controlling a PC

Remote Commander

Slide Show Commander

Command Post of the Future


remote commander
  • Allow PDAs to control a PC
    • Can be used with any application
    • Uses the standard (single) cursor
    • Don’t have to jump up andgrab mouse
    • All mouse and keyboard functions
      • Use PDA like touchpad
      • Graffiti or our own pop-upkeyboard
    • Word prediction and completion
pocketpc version
PocketPC version
  • Get PC’s screen onto PocketPC
  • Full view, or one-to-one zooming
    • Scroll with iPaq’s buttons
  • Control or scribble
for people with muscular disabilities
For People with Muscular Disabilities
  • Using handhelds as interface to PCs
    • People with Muscular Dystrophyhave fine-motor control but losegross motor control
      • Difficulties with mouse and keyboard,but stylus OK
  • So use Remote Commanderas PC’s keyboard and mouse
alphabet for accurate gestures
Alphabet for Accurate Gestures
  • Use a square hole in a template to increase accuracy
    • Just as easy to learnas Graffiti™
    • 18% more accuratefor able novices (p<.02)
    • Patent Pending
    • Also works with joystick
power point control
Power Point Control
  • Use PC to give the presentation
  • Use hand-held to control the PC
    • Two-way communication
  • Hand-held shows picture of slide,notes, timer
slideshow commander cont
SlideShow Commander, cont.
  • See list of slide titles
  • Scribble
  • Tap on on-screen buttonsand hyper-links
  • Preview otherslides
  • Controldemonstrations
control at a distance
Control at a Distance
  • Controlling a PC fromacross the room
    • Meetings, etc.
  • Use a laser pointer?
    • Studies
command post of the future
Command Post of the Future
  • Share media & control across multiple devices
    • Big displays, speech, gesture
    • And handhelds
cpof cont
CPOF, cont.
  • Provides remote control of main PC battleboard from handhelds
  • Also, private drill-down of public information
  • Small versions of map and data visualizations on handhelds
  • Interact locally, and displaychanges and annotationswhen ready
  • User-created panels of controls
  • Create custom interfaces and extensions to PC applications
    • And then take them with you
  • Direct manipulationfor edit, then setproperties
  • PocketPC or Palm
shortcutter widgets
Shortcutter Widgets
  • Buttons
  • Virtual Joy stick
  • Virtual Knob
  • Sliders
  • Mouse pad
  • Gesture panel
scenarios of use
Scenarios of Use
  • Lean-back mail reading
  • Controlling media players
  • … and many others
controlling appliances

Controlling Appliances

PhD research of Jeffrey Nichols

Assisted by undergrads and MS students

  • Too many complex devices, eachwith its own idiosyncratic interface
    • Stereo system
    • Telephones
    • ATM
    • Fax machine
    • Photocopier
    • Hotel alarm clock
  • Increasingly computerized
  • Low usability
existing universal controls
Existing “universal” controls
  • Pre-programmed at the factory with a subset,
  • Or, laboriously hand-programmed by the user
our approach
Our Approach
  • Handheld is a “Personal Universal Controller” (PUC)
  • Two-way communication
    • Appliance describes its functions
    • Handheld PUC
      • Automaticallycreates interface
      • Controls the appliance
      • Displays feedback about appliance status




automatic generation of uis
Automatic Generation of UIs
  • Benefits
    • All interfaces consistent for a user
      • With conventions of the handheld
      • Even from multiple manufacturers
        • Addresses hotel alarm clock problem
    • Can take into account user preferences
    • Multiple modalities (GUI + Speech UI)
  • A Hard Problem
    • Previous automatic systems have not generated high quality interfaces
research challenges
Research Challenges
  • Automatic Design of Graphical User Interfaces
  • Automatic Design of Speech User Interfaces
  • Connection with real devices
    • Through various protocols
    • X-10, AV/C, HAVi, UPnP, etc.
    • Also, custom hardware and software
hand generated graphical interfaces
Hand-Generated Graphical Interfaces
  • First, Hand-Designed PocketPC interfaces:
    • AIWA Shelf Stereo (Tape,CD,Tuner)
    • AT&T Telephone/AnsweringMachine
  • Used Embedded Visual Basic
  • Ensured quality with heuristicanalysis and think-aloud studies
  • Compared with manufacturer’sinterfaces
results of comparison
Results of Comparison
  • Using PUC, users took 50% less time & made 50% fewer errors
  • All differences are significant (p < 0.05)
current puc specification language
Current PUC Specification Language
  • XML
  • Full documentation for the specification language and protocol:

    • Contains sample specification for a stereo
properties of puc language
Properties of PUC Language
  • State variables & commands
    • Each can have multiple labels
      • Useful when not enough room
    • Typed variables
      • Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.
    • Optional labels for values
  • Hierarchical Structure
    • Groups
dependency information
Dependency Information
  • Crucial for high-quality interfaces
  • Expressed as <active-if> clauses
    • Operations:
      • Equals, Less-Than,Greater-Than
    • Combined Logically
      • AND, OR
  • Used for:
    • Dynamic graying out
    • Layout
    • Widget selection
  • Have working specifications for:
    • Audiophase stereo
    • X-10 lights control
    • Sony CamCorder
    • Windows Media Player
    • Audio ReQuest hardware MP3 player
    • WinAmp Media Player
    • Elevator
    • Parts of GMC Yukon Denali SUV
    • Etc.
controller generators
Controller Generators
  • iPaq PocketPC
  • SmartPhone
    • No touchscreen
  • Desktop (TabletPC)
  • Speech
examples of generated guis
Examples of Generated GUIs

Shelf stereo Windows Media Player

generating speech interfaces
Generating Speech Interfaces
  • “Universal Speech Interface” (USI) project
    • Prof. Roni Rosenfeld of CMU
  • Creates grammar, language model and pronunciation dictionary from PUC specification
    • Pronunciation from labels using phonetic rules
    • Can provide other pronunciations as labels for fine-tuning
  • Will use dependency information to help with disambiguation and explanation
  • Supports queries and spoken feedback
    • Paraphrases as confirmation
“Adaptors” provide the interface to existing (and future) appliances

If do not support specification language directly

Custom hardware

Custom software


Windows Media Player


Light switches, etc.

AV/C (standard protocol)

Sony CamCorder



Axis Camera

  • Sony CamCorder
  • Windows Media Player
  • Image library
  • Two way communication:
    • When state changes from application or appliance, GUI is updated
    • Can query state with speech
current work on puc
Current Work on PUC
  • Controlling User “Experiences”
    • One command for multiple devices
    • Play a DVD  sets DVD, cable, TV
    • Setting alarm clock  sets heat, coffee, car
  • Consistency for Users
    • And user preferences
  • More devices
    • UPnP
    • Etc.
for more information
For More Information
  • Many papers on the Pebbles web site
    • On PUC and other work
  • Most programs available for free downloading:
  • SlideShow Commander islicensed for commercial sale
    • See

  • As more and more electronics are computerized and communicating, mobile devices can help control them
  • Handheld devices can improve the user interfaces of everything else
    • Not just be another gadget to be learned
thanks to pebbles students
Thanks to Pebbles Students
  • This research performed by a large number of students since 1997:
thanks to our sponsors
The Pebbles research is supported by grants from:




General Motors

NEC Foundation

Pittsburgh Digital Greenhouse

And equipment grants from:

Hewlett Packard

Lucent Technologies

Lantronix, Inc.

Lutron, Inc.


Palm Computing

Symbol Technologies


SMART Technologies


Synergy Solutions


Thanks to Our Sponsors!
mobile devices for control of ubiquitous multimedia47

Human Computer Interaction Institute

School of Computer Science

Carnegie Mellon University

Thank You!

Mobile Devices for Control ofUbiquitous Multimedia

Brad A.