using hand held computers and pcs together the pebbles project
Skip this Video
Download Presentation
Using Hand-Held Computers and PCs Together: The Pebbles Project

Loading in 2 Seconds...

play fullscreen
1 / 64

Using Hand-Held Computers and PCs Together: The Pebbles Project - PowerPoint PPT Presentation

  • Uploaded on

Human Computer Interaction Institute School of Computer Science Carnegie Mellon University. Using Hand-Held Computers and PCs Together: The Pebbles Project. Brad A. Myers [email protected] Multiple Devices.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Using Hand-Held Computers and PCs Together: The Pebbles Project' - kenny

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
using hand held computers and pcs together the pebbles project
Human Computer Interaction Institute

School of Computer Science

Carnegie Mellon University

Using Hand-Held Computersand PCs Together:The Pebbles Project

Brad A. Myers

[email protected]

multiple devices
Multiple Devices
  • Most of our time spent in places where there is embedded technology
    • Offices, meeting rooms, classrooms, homes
  • Often multiple devices will be available:
    • Mobile phone and PDA and a PC
    • Multiple people’s PDAs
    • PDAs in a “Smart Room”
handhelds will be communicating
Handhelds will be communicating
  • 802.11
  • BlueTooth
  • Cell-phone network
  • (Infrared)
premises of our research
Premises of our Research

“With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be inclose, interactive communication.”

  • Brad Myers, “Using Hand-Held Devices andPCs Together,” Comm. ACM,Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
research agenda
Research Agenda

How can multiple devicesbe used effectively together,at the same time?

How can the user interface and functionality be spread across multiple devices?

research agenda6
Research Agenda

“Multi-MachineUser Interfaces”

How can multiple devicesbe used effectively together,at the same time?

How can the user interface and functionality be spread across multiple devices?

example power point control
Example: Power Point Control
  • Use PC to give the presentation
  • Use hand-held to control the PC
    • Two-way communication
  • Hand-held shows picture of slide,notes, list of titles, timer, etc.
pebbles is
Pebbles is:

DAs for

ntry of


ytes and

ocations from










handhelds in an office
Handhelds in an Office
  • Augment desktop applications
  • Use multiple devices at the same time
handhelds in meetings
Handhelds in Meetings
  • Attendees use handhelds to interact andannotate presentation
  • Augment collaboration
  • Take notes
handhelds in classrooms
Handhelds in Classrooms
  • Data projectors for instructor’s slides
  • Students could have computers for:
    • Notetaking linked to instructor’s slides
    • In-class testing
    • Running simulations
handhelds in homes
Handhelds in Homes
  • Interact with embedded computation
    • “Smart homes”
    • Not just speech and vision as interfaces
  • Interact with appliances, lights, etc.
    • “Personal Universal Controller” (PUC)
office use for individuals

Office Use (For Individuals)

How can handhelds augment desktop applications?

applications for individuals
Applications for Individuals
  • Extra input and output devices have been shown to be useful
  • But can be expensive and hard to configure
  • People have PDAs and are attached to PC
    • For example, cradles for recharging
  • Customizable, extensible
  • Extend desktop applications
scrolling with the pda
Scrolling with the PDA
  • For scrolling using the non-dominant hand
    • Studies showed parallel and efficient uses of both hands together
    • Generates Windows scrolling events
results of study of scrolling with pda
Winner!Results of Study of Scrolling with PDA
  • Scrolling with buttons on PDA was fastest
  • PDA scrollers similar to mouse speed
  • Using 2 hands is effective!
remote clipboard
Remote Clipboard
  • Transfer information between PDA and PC
  • Connects their clipboards together
  • Transfer content or reference
  • Works with all applications
  • Also between multiplecomputers
  • User-created panels of controls
  • Create custom interfaces and extensions to PC applications
    • And then take them with you
  • Direct manipulationfor edit, then setproperties
  • Palm or PocketPC
shortcutter widgets
Shortcutter Widgets
  • Buttons
  • Sliders
  • Knobs
  • Mouse pad
  • Graffiti Pad (Palm)
  • Gesture panel
shortcutter actions
Shortcutter Actions
  • Send any keyboard key, mouse button, scrolling action or string to PC
  • Open a file or URL
  • Run an application
  • Invoke any PC menu or button
    • Windows message
    • Recorded
  • Switch to a different Shortcutter panel
  • Control the Mouse
shortcutter actions cont
Shortcutter Actions, cont.
  • Control external devicesthrough PC’s serial port
    • Directly (e.g., projectors)
    • X-10 for electrical devices
  • Macro
    • Can be multi-application
  • Application-specific
    • Same button, different messages
    • Useful for application sets:browsers, compilers
more scenarios of use
More Scenarios of Use
  • Lean-back mail reading
  • Controlling WinAmp
  • … and many others
study of individual use
Study of Individual Use
  • Time to tap on button depended on size
  • Few errors
  • People often didn’t look at PDA
study of individual use24
Study of Individual Use
  • Moving hands to both PDA and mouse only about 15% slower than just moving to the mouse


1H Keyboard->Mouse


1H Keyboard->PDA


1H Mouse->Keyboard


% slower

1H PDA->Keyboard


Keyboard -> Mouse&PDA



Mouse&PDA -> Keyboard



use in meetings

Use in Meetings

Enhance group’s collaboration and control

original application remote commander
Original Application:Remote Commander
  • Allow PDAs to control a PC
    • Can be used with any application
    • Uses the standard (single) cursor
    • Don’t have to jump up and grabmouse
    • Perform all mouse andkeyboard functions
      • Use PDA like touchpad
      • Graffiti or our ownpop-up keyboard
pocketpc version
PocketPC version
  • Get PC’s screen onto PocketPC
  • Full view, or one-to-one zooming
    • Scroll with iPaq’s buttons
  • Multiple people draw on top of whatever on PC screen, not just PowerPoint
  • Each user has own cursorand color
  • Save by PrintScreen
  • Erase by refresh
slideshow commander
SlideShow Commander
  • For PowerPoint
    • PC Only: Windows 95, 98, NT, 2000, ME, XP
  • Use PC to give the presentation
    • Full features of PowerPoint
  • Use hand-held as “remote control” for PC
    • Can wander away from the keyboard
    • Two-way communication
  • View, Navigate, Highlight, Time
view on hand held slide
View on Hand-Held: slide
  • See thumb-nail of current slide
    • Black and white or color
view on hand held notes
View on Hand-Held: notes
  • See the notes of the current slide
view on hand held titles
View on Hand-Held: titles
  • See the list of titles
view on handheld time
View on handheld: Time
  • Multi-function timer
  • View large
  • And in corner
control demonstrations on palm
Control Demonstrations on Palm
  • Control PC applications and external devices
  • Uses Shortcutter
  • Easy demo and resume show
control demonstrations on pocketpc
Control Demonstrations on PocketPC
  • Task list of PC applications
  • Tap brings one to front
  • PowerPoint continuesbehind – easy to resume
  • SlideShow Commandercommercialized by:
  • Available at Office Depot,CompUSA, etc.
  • Palm and PocketPC

Synergy Solutions, Inc.

studies of presentations
Studies of Presentations
  • Summer study of 2 HCI Master’s students
  • Contextual Inquiry of 9 presentations
    • Audience: 10 to hundreds
    • 7 used PowerPoint, 4 used NetMeeting
  • Found 220 “breakdowns”
    • Most were minor problems
    • Averaged 8.7 lost minutes per talk (14.5%)
  • Designed SlideShow Commander to eliminate some of these problems
  • For special applications that are awareof multiple inputs
  • E.g.: Shared Whiteboard
  • Single Display Groupware
    • Multiple people, one display
    • A number of interestingissues:
      • Palettes, widgets
      • Section handles
      • Undo
  • Communicate to another PDA user through the PC
  • PC serves as a conduit
  • For side notes and messages
    • For example, in negotiationmeetings
  • Send to all or to a specificperson
command post of the future
Command Post of the Future
  • Large DARPA funded project
  • Make commanders more effective
study of laser pointers
Study of Laser Pointers
  • Studied properties of laserpointer interaction techniques
  • Hand-wiggle 8 pixels
  • Delay until moving average stable 1.5 sec
  • Delay until target acquired 1.5 sec
  • Different devices and poses do not help much
tap speed study
Tap Speed Study
  • Tapping directly onSmartBoard is fastest
  • Semantic Snarfing next
    • But high error rate
  • Then regular mouse
  • Laser pointing slowest
    • Used separate physicalbutton on handheld
semantic snarfing
“Semantic Snarfing”
  • Interacting at a distance
  • Grab contents to handheld
    • Picture
    • Menus
    • Text
    • Re-visualization
      • “Magic Lenses”
classroom use

Classroom Use

Using handheld computers in classrooms

For testing

Improve large lectures

Provide immediate feedback to instructor

  • HP donated 110 Jornada 680 and 100 720 computers
    • Windows CE
  • Lucent donated Wavelan wireless cards
  • Collaborator: Prof. David Yaron of Chemistry
  • Chemistry 09-106: Modern Chemistry II
    • Spring, 2000 and Spring, 2001
    • About 90 students
    • Mostly freshmen
  • Loaned one Jornada and Wavelan card to each student for the whole semester
    • So would get used to using it
    • Offset technical difficulties with benefits to student
wireless andrew
Wireless Andrew
  • CMU has almost complete coverage of 802.11b support
  • Wavelan
    • Donations from Lucent, etc.
    • Funding from state of Pennsylvania, etc.
  • Can use computers wirelessly in all classrooms and offices
    • Most lawns, dorm rooms, etc.
  • Windows, Mac, Linux, Windows CE support
pebbles tests
Pebbles Tests
  • Create through html pages
  • Arbitrary html in questions and answers
    • Pictures, formatting, links
  • Use FrontPage, etc. to author
  • Embed tags to show question and answers
concept test
Concept Test
  • Students allowed to answer multiple times
    • Server stores all answers
  • Prof. Yaron displayedquestions and answers onthe board
  • Evidence that significantlycontributes to learning
    • Mazur, E. (1997).Peer Instruction:A User's Manual, Prentice Hall.
instructor s view
Instructor’s View
  • In class, instructor sees a chart of answers
    • Optionally refreshes every 3 seconds
  • May be projected forwhole class tosee
  • Top displaysinstructions forstudents
  • Unfortunately, no data (yet) about effects on learning
  • Students thought concept tests were a valuable part of the class
  • Students significantly preferred using handhelds for concept tests.
home use

Home Use

Personal Universal Controller

Help for people with muscular disabilities

personal universal controller
Personal Universal Controller
  • Two-way communication
    • Appliances describe their functions
    • Handheld PUC creates interface based on descriptions
    • Handheld PUC controls the appliance
    • Appliance sends back status as feedback



Feedback of Status

initial experiments
Initial Experiments
  • Use real PocketPC
    • Pretend that controls devices
pocketpc results
PocketPC results
  • For both appliances, users of actual interfaces:
    • Took about twice as long
    • Made at least twice as many mistakes as users of the handheld interfaces
    • Needed external help five times more often
current steps
Current Steps

Specification Language



name="Audiophase 5 CD Stereo">


  • XML specification language and protocol for describing appliances
  • Create panels automatically from the specification
  • Find realappliances whichwe can control
current work handhelds for people with muscular disabilities
Current Work: Handhelds forPeople with Muscular Disabilities
  • Using handhelds as interface to PCs
    • People with Muscular Dystrophyhave fine-motor control but losegross motor control
      • Difficulties with mouse and keyboard,but stylus OK
  • Handhelds as interface to other devices
    • Control room lights, telephone, wheelchair, etc.
remote commander adaptations
Remote Commander Adaptations
  • Faster Acceleration
  • More flexible tapping
  • Turn off key repeat
  • Multiple Keyboards
general architecture
General Architecture

On the PC

Any PC app.

Brad’sPalm Pilot



Rob’sPocket PC




Serial,IR orsockets

Ben’sWindows CE

Direct connection or sockets




Various PDA apps

Various dlls

  • Can use Pebbles protocols to develop your own application
    • Libraries for Palm, Windows CE, PC
    • Independence from communication medium
  • Use by:
    • PalmAmp from IronCreek Software
    • Intel research
    • others…
  • Windows messages or sockets
  • Most of this software is available for free downloading
  • First release in Feb, 1998
  • Just released version 5 (Feb, 2002)
  • Downloaded over 30,000 times
    • About 200 times a week

  • Handhelds are becoming ubiquitous
  • Connecting technologies will improve
  • Important to study how can be used when connected to computers and each other
    • “Multi-Machine User Interfaces” will be increasingly important
  • People will want to use the most convenient device for their information and control
thanks to our sponsors
Supported by grants from:



Pittsburgh Digital Greenhouse


And equipment grants from:

Hewlett Packard

Lucent Technologies

Palm Computing

Symbol Technologies


SMART Technologies, Inc.

Synergy Solutions, Inc.


Thanks to Our Sponsors!
using hand held computers and pcs together the pebbles project64
Thank you!

Human Computer Interaction Institute

School of Computer Science

Carnegie Mellon University

Using Hand-Held Computersand PCs Together:The Pebbles Project

Brad A. Myers

[email protected]