Interaction gestural input popup widgets interaction techniques and hardware input devices
Sponsored Links
This presentation is the property of its rightful owner.
1 / 97

Interaction: gestural input, popup widgets, interaction techniques, and hardware input devices PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Interaction: gestural input, popup widgets, interaction techniques, and hardware input devices. Prof. Michael McGuffin. Gestural Input. Gestures can be executed rapidly, and can be used as symbols to activate commands or select objects Sketches

Download Presentation

Interaction: gestural input, popup widgets, interaction techniques, and hardware input devices

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Interaction: gestural input, popup widgets, interaction techniques, and hardware input devices

Prof. Michael McGuffin

Gestural Input

  • Gestures can be executed rapidly, and can be used as symbols to activate commands or select objects

  • Sketches

    • can be used to quickly enter text or diagrams, without requiring the user to switch from the pointing device to something else

    • Have an informal, loose, implicit structure

    • Can carry much more (implicit) information than typed text

  • Gestures as characters to recognize

    • Graffiti, Unistroke

Gestural Input

Kurtenbach and Buxton (video)

  • Example gestures: circle objects, drag to move, or drag + draw a "C"  to copy

  • Such a gesture selects the noun (objects) and verb (operation) in a single action, which may be more natural, fluid, or faster than doing selection + Ctrl-C/Ctrl-X + Ctrl-V

Gestural Input

"Teddy" (Igarashi et al. 1999).

See Java applet at Takeo Igarashi’s website.

Phun (≈2008)

  • A free game/physical simulator that uses gestural input



  • Alvarado and Davis 2001


  • LaViola 2007


  • Most user interfaces that use gestural input require the user to first learn the gestures before they can be used. This can be a significant obstacle for novice users.

  • In the following slides, we will see a technique called Marking Menus that eliminates this problem.

Modes, and some examples of interaction techniques using gestural input or other input

Interaction Techniques

  • “are ways to use input devices to enter information”

  • "are made up of single input-device actions.”

    (Foley, et al. “Computer Graphics: Principles and Practice”)



Verbs in a pulldown menu


Verbs ina popup menu



  • Modes can lead to mode errors, where the user thinks they are in one mode but are really in another

  • Visual feedback (example: changing cursor shape) to indicate the current mode is good, but often not enough to avoid mode errors

  • Popup menus help…

    • Avoid mode errors(via transient modes, and kinaesthetic feedback where the user feels pressure through their finger from holding down a button)

    • Increase screen space devoted to content, because the menu is only displayed when desired

    • Decrease the distance travelled by the cursor, because there is no back-and-forth motion between the workspace and peripheral interface elements

    • Create a tighter integration, or fusion, of the selection of noun and verb, leading to better mental chunking (Buxton 1986)

Given that popup menus have all these advantages, can we design other widgets that are even better?

Radial Menu (or “Pie Menu”)

Demo: Radial Menu in SimplePaint

An interface that uses, effectively, radial menus

Yatani et al., CHI 2008

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Radial menusversus linear menus

Directions are easier to rememberand reproduce than distances

Hierarchical Radial Menu

"Mouse Gestures" for Firefox

Marking Menu

  • “Scale invariant recognition”: The recognition of the gestures (marks) does not depend on the length of the segments; only the angles matter. Hence, the marks can be drawn small and quickly, or ballistically.

  • A user who knows what mark to draw doesn’t even need to see the menu popup.

The set of gestures is discoverable and “self-revealing”, contrary to other gestural interfaces

Improved graphical presentation

Marking Menus

  • Video/Demonstration

Marking Menus


Marking Menus


Transition from novice to expert

Marking Menus:Transition is gradual and natural !

Traditional menus:Pointing versus shortcuts


Marking Menus

  • Can be operated faster than linear menus(ballistic, directional, scale-invariant marks)

  • Have a self-revealing gesture set

  • Enable a smooth transitionfrom novice use to expert use

  • Can be used to select both a noun and verb

  • Are limited to 8 items per submenu,and limited to a depth of about 3 levels

Marking Menus

  • Allow for faster selection than with linear menus(directional, ballistic movements)

  • Can be used without looking at the screen(“eyes-free operation”)

  • Have a discoverable set of gestures

  • Allow for a gradual and natural transition from novice to expert

  • Can be used to select a noun and verb

  • Are limited to approximately 8 items per submenu, and a depth of about 3 levels


  • Temporary modes, held by holding down a button or key with kinesthetic feedback …

    • Help avoid mode errors

    • Example: a key that must be held down to maintain a temporary mode change, such as the Shift key on keyboards

    • Example: popup widgets

      • Help combine noun+verb selection

      • Only require screen space when in use

  • One popup widget with many advantages: Marking Menus

    • Allow for ballistic gestures

    • Allows users to gradually transition from novice to expert

Other examplesof modes, tools, etc.

  • video: "Selection and Positioning tasks", Bill Buxton 1983

Some observations

  • Techniques shown in the video:

    • 1. Drag-and-drop

    • 2. Modal toolbar

    • 3. « Moving menu » (a kind of popup menu)

    • 4. « Moving menu » with memory

  • Questions: of the 4 techniques, which ones …

    • Are not modal (or only have a temporary mode with kinesthetic feedback), and so help to avoid mode errors ? (Answer: 1 and 3)

    • Avoid back-and-forth movements between the workspace and the toolbar, and do not have screenspace consumed by a toolbar ? (Answer: 3 and 4)

    • Have visible affordances (toolbar), therefore showing the possible operations to a novice user ? (Answer: 1 and 2)

    • Allow the same shape to be created many times repeatedly, very quickly ?(Answer: 2 and 4)

    • Ensure that each shape is always selected with the same dragging gesture, allowing the user to learn the gestures by heart and then execute them rapidly ? (Answer: 1 and especially 3.)

Can we extend popup menus/widgetsfor other uses?

Other popup menus and widgets

  • Hotbox (Kurtenbach et al., 1999),Control Menus (Pook et al., 2000),Flow Menus (Guimbretière et Winograd, 2000),FaST Sliders (McGuffin et al., 2002),Tracking Menus (Fitzmaurice et al., 2003),Trailing Widget (Forlines et al., 2006)Hover Widgets (Grossman et al., 2006),PieCursor (Fitzmaurice et al., 2008)

  • These widgets and interaction techniques are designed for:

    • A large number of commands

    • Controlling continuous variables

    • Input of text and numbers with gestures

    • Use with a stylus (pen), for example, on a “tablet PC”

FlowMenus (Guimbretière et al., 2000)

2D manipulation with FlowMenus(Guimbretière et al., 2000)

The Hotbox in Maya: a 2D menu

[Kurtenbach et al., 1999]

The Hotbox in Maya: a 2D menu

[Kurtenbach et al., 1999]

The Hotbox in Maya: a 2D menu

[Kurtenbach et al., 1999]

Demo of a more recent kind of Hotbox (McGuffin and Jurisica 2009) used for manipulating a network visualization

Tracking Menu (copy of video at George Fitzmaurice’s website)

Hover Widgets (copy of video at at Ravin Balakrishnan’s website)

Toolglass: bimanual input

Click-through: Simultaneousselection of verb and noun!

Bimanual 3D Navigation

Erick Velazquez-Godinez and Michael McGuffin (unpublished, 2010) the 3D position of each hand is enough to control the 6 degrees-of-freedom of a 3D camera: translate hands together to translate the camera, rotate hands around each other to rotate the camera, move hands toward or away from each other to zoom.

Tracking 2 or more fingers with a Nintendo Wiimote

  • See online videos by Johnny Chung Lee

Other uses of two hands?

Two-Handed (Bimanual) Input

Potential uses:

  • Dominant hand (DH) on mouse,non-dominant hand (NDH) on keyboard

  • Two mice, two cursors, symmetrical:

    • Rapid clicking by alternating between hands?

    • Simultaneous rotation+scaling+positioning in 2Dor rotation+zooming+panning in 2D

  • Two mice, asymmetrical:

    • NDH for camera, DH for selection/manipulation

    • NDH for tool palette, DH for clicking-through (Toolglass)

Kinematic Chain Model(Yves Guiard 1987)

  • NDH (non-dominant hand) performs slower and less precise movements than the DH (dominant hand)

  • The NDH moves before the DH

  • The NDH establishes a frame of reference within which the DH performs work

Input Devices


Foldable keyboard for Palm Pilot

Flexible keyboard; 24 $ at from 2008)

Projected Keyboard

160$ at of 2008)


Optimus Maximus

Each of the 113 keys contain a 48x48 pixel screen.

1864 $ US ! (price of 2008)

Optimus Maximus

Configured for English

Optimus Maximus

Configured for Russian

Optimus Maximus

Configured for the game "Half-life"

Optimus Maximus

Configured for "Photoshop"

Bottons on a tape recorder

The status quo:

Uniform buttons

Buttons with different forms: allow use without looking (“eyes-free operation”)

modern design

naïve design

Piano Keyboard

Apple iPhone Keyboard

Scholes Keyboard versus Dvorak Keyboard

Still not in widespread use

Notice that all vowels are under one hand in the midlde row.

Designed in 1873 !

Still the standard :-(

Comparison of QWERTY and Dvorak

“Half Keyboard” from Matias Corp.

« Septambic keyer »or « chorded keyboard »

4 x 24 – 1 = 63 possible "chords"

Pointing Devices

The first mouse

  • 1968

  • Douglas Engelbart

  • Stanford Research Institute

  • Two rollers for x and y

40+ years later

Rotation sensing

Rockin’ Mouse

Haptic feedback(or “force feedback”)


Other kinds of 2D pointing devices

  • Digitizing tablet / graphics tablet with a stylus (pen) and/or puck (mouse)

  • Touchscreen

  • Light pen

  • Touchpad

  • Joystick

    • isometric (rigid, senses pressure, for example: Trackpoint)

    • elastic (like isotonic, but returns to center when released)

    • isotonic (can be moved freely)

  • Trackball

  • Eye tracker

TrackPoint (isometric joystick)


Logitech Trackball

Graphics tablets

Wacom Bamboo

5.8x3.7 inches; 80$

Wacom Intuos3

12x19 inches; 750$

Wacom Cintiq 21UX

with integrated screen

21.3 inches (17x12.75); 2500$

(prices of 2008)

Input devices for tablets

  • Styluses can have

  • a pressure sensor on their tip

  • a button on the tip to click

  • a button on the side ("barrel button")

  • a scroll wheel that can be rotated

IntuPaint (Vandoren et al. 2008)and FluidPaint (Vandoren et al. 2009)

Properties of Pointing Devices

Absolute vs relative sensing

Which is more general ?

Direct vs indirect

Which is more “intuitive” ?

Which is more predictable ?

Which is more relaxing ?

Which leaves visual feedback more visible ?

Which do artists prefer ?

Discrete vs continuous

Which is more general ?

Position control vs velocity control

Which is more general ?

Which typically requires less movement/smaller footprint ?

Velocity control:

Velocity control pointing devices, for 3D


Spaceball 5000



SpaceNavigator (60$)

(in 2008)

Other products from

(in 2008)

Taxonomy of input devices (Buxton)

M: Mechanical intermediary

T: Touch

3-State Model (Buxton, 1990)

State 0: no (x,y) coordinates

States 1 and 2: (x,y) position is known

  • Examples:

  • Graphics tablet: states 0, 1, 2

  • Mouse: states 1, 2

  • Touchscreen: states 0, 1

TouchMouse(Hinckley and Sinclair 1999)

States 0, 1, 2

TouchMouse(Hinckley and Sinclair 1999)

“Pop Through” Buttons(Zeleznik et al. 2001,

PreSence (Rekimoto et al. 2003)

Haptic Pen

Lee et al., "Haptic pen: a tactile feedback stylus for touch screens", UIST 2004

Handheld Projector and Pen

Cao et Balakrishnan, "Interacting with Dynamically Defined Information Spaces using a Handheld Projector and a Pen", UIST 2006


Song et al., "PenLight: combining a mobile projector and a digital pen for dynamic visual overlay", CHI 2009

  • Login