High performance pen interfaces
Download
1 / 79

- PowerPoint PPT Presentation


  • 129 Views
  • Updated On :

Ken Hinckley Microsoft Research October 22 nd , 2004 [email protected] http://research.microsoft.com/~kenh. High Performance Pen Interfaces. Pen Computing Now?. Project #1: Stitching. Ken Hinckley Gonzalo Ramos Francois Guimbretiere Patrick Baudisch Marc Smith. Wireless Networks.

Related searches for

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about '' - piper


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
High performance pen interfaces l.jpg

Ken Hinckley

Microsoft Research

October 22nd, 2004

[email protected]

http://research.microsoft.com/~kenh

High Performance Pen Interfaces



Project 1 stitching l.jpg
Project #1: Stitching

  • Ken Hinckley

  • Gonzalo Ramos

  • Francois Guimbretiere

  • Patrick Baudisch

  • Marc Smith


Wireless networks l.jpg
Wireless Networks

  • Still an immature technology –

    • Flaky, hard for users to configure, slow, …

    • Most research has focused on small-screen presentation of web pages

    • What are the completely new & unique affordances of wireless connectivity?

  • Odd phrase: not what it is, but what it lacks


Wireless network 2004 horseless carriage 1904 l.jpg
Wireless Network 2004 = Horseless Carriage 1904?

  • Are Wireless Networks really just wired networks without the wires?

  • Or are they something completely different?


Wireless device soup which links does the user want l.jpg
Wireless Device Soup: Which links does the user want?


Name that device l.jpg

Users need techniques to intuitively form bridges between devices

How do users name the devices to connect?

What is type / purpose of the connection?

Parameters? (Who copies what, to where?)

Name That Device


Stitching l.jpg

Pen stroke devicesthat spans displays

Move the pen

Cross over bezel

Finish stroke on nearby tablet

System infers connection

Stitching


Central design questions l.jpg

How is a connection established? devices

What type of connection is required?

What information is shared?

What is the spatial relationship between devices?

How do users share physical space?

Explored in context of photo sharing app

Central Design Questions


Video l.jpg

AVI’04 devices

ACM Advanced Visual Interfaces 2004

VIDEO


1 establishing a connection l.jpg

Recognizes timing & dynamics of pen trajectory devices

There is nothing special about the pen!

Wireless signal strength determines nearby devices

Tablet 2

Tablet 1

Δt

1. Establishing a Connection


2 type of connection l.jpg

User selects a devicesmulti-device command

e.g., Pie menu at end of stitching gesture

2. Type of Connection


3 what information to share l.jpg

Many commands may need operands devices

Lasso to select, continue to other display to stitch.

3. What Information to Share?


4 what is spatial relationship automatic screen calibration l.jpg

Fits a line to absolute coords of user’s pen stroke devices

4. What is Spatial Relationship: Automatic Screen Calibration


5 how do users share physical space l.jpg

Sociology devicesProxemics: How people use space

Invisible bubble that surrounds an individual

5. How Do Users Share Physical Space?


6 how do users share physical space l.jpg

Sociology devicesProxemics: How people use space

Invisible bubble that surrounds an individual

6. How Do Users Share Physical Space?

  • Homework assignment:

    • Sit right next to someone at airport (when it is not necessary to do so)

    • Time how long it is before the other person leaves


5 how do users share physical space17 l.jpg

Sociology devicesProxemics: How people use space

Invisible bubble that surrounds an individual

5. How Do Users Share Physical Space?


Proxemic lessons for stitching l.jpg

Don’t devicesrequire contact : touching is taboo

“Do they have to be right next to one another?”: 10-40cm

“wide screen would be nice for collaboration”

“no two faces trying to peek at only one screen”

Proxemic Lessons for Stitching


Proxemic lessons for stitching21 l.jpg

Don’t devicesrequire contact

“Do they have to be right next to one another?”: 10-40cm

“wide screen would be nice for collaboration”

“no two faces trying to peek at only one screen”

Gives users flexibility to be involved, or not: mood, task, …

Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague

Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc.

Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach

Proxemic Lessons for Stitching


Proxemic lessons for stitching22 l.jpg

Don’t devicesrequire contact

“Do they have to be right next to one another?”: 10-40cm

“wide screen would be nice for collaboration”

“no two faces trying to peek at only one screen”

Gives users flexibility to be involved, or not

Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague

Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc.

Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach

Support varying relative body orientation

Proxemic Lessons for Stitching


Proxemics orientation of spaces l.jpg

6 devicesconversations

3

1

0

Proxemics: Orientation of Spaces

  • Communication patterns change wrt orientation

  • Stitching supports:

    • Shoulder-to-shoulder

    • Face-to-face

    • At 90 angles

Relative frequency:


Ongoing work l.jpg

Multiple Devices: devicesCooperative Stitching

User 1

User 2

User 4

User 3

Ongoing work


Ongoing work25 l.jpg

Multiple Devices: devicesCooperative Stitching

User 1

User 2

User 4

User 3

Ongoing work


Project 2 scriboli l.jpg
Project #2: devicesScriboli

  • Ken Hinckley

  • Patrick Baudisch

  • Gonzalo Ramos

  • Francois Guimbretiere


Round trip problem on tablet pc l.jpg
Round trip problem on Tablet PC devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem29 l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem30 l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem31 l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem32 l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem33 l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem34 l.jpg
Round trip problem devices

  • 4 trips (+ some xtra) for copy/paste


Round trip problem35 l.jpg
Round trip problem devices

  • No right click, kbd shortcuts to avoid this!


Gedit kurtenbach buxton l.jpg
GEdit – devicesKurtenbach & Buxton

Delete one obj.

Copy – add final “C” to move

Delete group – end stroke inside lasso

  • One of the first works considering pen phrases

  • Every stroke is gesture

  • Only 3 cmds

  • Group vs. single obj. inconsistent

Move – end stroke outside. Objects move on PenUp.


Sketch brown university l.jpg
SKETCH – Brown University devices

  • Adding 1 more gesture breaks everything

  • Specific to 3D, difficult to learn how to use


Marking menus l.jpg
Marking Menus devices

  • Prompted BlindScale independence

  • Self-revealing: prompts w/ available cmds

  • Rapid expert use: blind marking

    • Menu appears only after 1/3 second delay

    • Size of mark does not matter


Flowmenu francois guimbretiere l.jpg
FlowMenu – devicesFrancois Guimbretiere

  • Integrates command selection with continuous parameter manipulation

  • But no “blind marking,” size of gesture matters

Crossing inout chooses Item… submenu

Crossing outin chooses Move command

Dragging continues


Tivoli l.jpg
Tivoli devices

  • Tivoli project @ Xerox PARC

  • We wanted quick, informal feel of scribbling


Learning cursive l.jpg
Learning devicesCursive

  • building blocks of writing as basis for pen UI?

  • not natural handwriting recognition


The deep structure of handwriting l.jpg
The Deep Structure of Handwriting devices

  • The “scribble” of Scriboli

  • Phasic muscle contractions:

    • fast, brief, usually repetitive

    • Pre-programmed, internally driven

    • “The conversion of those […] O’s to real words and sentences happens when small variations in the amount of verticality or laterality […] are introduced into the movement. The extreme orderliness and predictability of individual handwriting is due to this mode of generating and controlling the movement.”F. R. Wilson, The Hand, p. 163


Design philosophy of scriboli l.jpg
Design Philosophy of devicesScriboli

  • RAPID

    • Fast motions, repeatable, no waiting / dwelling, minimal demands on visual attention

  • UNAMBIGUOUS

    • No guessing games

    • No reco unless I tell the system otherwise

    • (There is almost no reco in Scriboli)

  • EXPRESSIVE

    • Supports many commands, easy to grow design

    • Mechanisms for a wide span of pen apps, not restricted to specific niche application


Grammar for pen input l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separates words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope (lasso, line, …)

Marking menu

Crossing manip. phase

Grammar for pen input

  • Scriboli proposes fundamental building blocks

  • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases


Ink vs gesture l.jpg
Ink vs. Gesture devices

  • Who does the recognition?

    • Ink: strokes intended primarily for human

    • Gesture: strokes intended for computer

  • Virtually all pen systems make this distinction at some level

  • Very seductive to attempt to do automatically

    • Many systems spend most or all effort just on this

    • Any pen gesture might occur in freeform input

  • Scriboli uses explicit ink/gesture mode switch


Ink gesture study w u washington l.jpg
Ink/Gesture study devicesw/ U. Washington

  • 5 Techniques:

    • Barrel Button

    • Tap+hold

    • NPH button

    • Pen Pressure

    • Pen Eraser


Grammar for pen input47 l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separates words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope (lasso, line, …)

Marking menu

Crossing manip. phase

Grammar for pen input

  • Scriboli proposes fundamental building blocks

  • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases


Delimiters single stroke lasso selection marking l.jpg
Delimiters: Single-stroke deviceslasso selection + marking

  • Initial part of stroke is the scope

  • Pigtail is the delimiter

  • Tail interpreted as the mark


What is a delimiter l.jpg
What is a Delimiter? devices

  • A delimiter is “something different” about the pen input stream

    • Determines lexical structure of pen stroke

  • Pigtail only intended as delimiter during Gesture mode

    • Not an automatically inferred mode switch


Grammar for pen input50 l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separates words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope(lasso,line, …)

Marking menu

Crossing manip. phase

Grammar for pen input

  • Scriboli proposes fundamental building blocks

  • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases


Direct manipulation phase l.jpg
Direct manipulation devicesphase

  • Copy-paste in place

    • Zero round trips

  • Continuing past outer border turns into dragging (like FlowMenu)

  • A single stroke supports:

    • Scope

    • Command activation

    • Dragging (direct manip)

  • FlowMenu cannot support all 3


Direct manip phase preserves scale independence l.jpg

Menu activation devices

visual menu pops up

cross boundary

check for crossing

dragging

blind marking

+333 ms

Direct manip phase preserves scale-independence

  • Possible to have both:

    • boundary crossing (like FlowMenu)

    • scale-independence (like marking menus)

  • Scale-ind. only matters for blind marking

  • Dragging naturally lasts longer than 1/3 second.


Grammar for pen input53 l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separates words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope (lasso, line, …)

Marking menu

Crossing manip phase

Grammar for pen input

  • Scriboli proposes fundamental building blocks

  • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases


Scriboli video l.jpg
Scriboli Video devices

  • VIDEO


Delimiters performance study l.jpg
Delimiters performance study devices

  • Pigtail

  • Timeout

    • drag+hold during lasso

  • Multi-stroke

    • acquire selection handle after lasso

  • Button

    • Hit button during lasso to delimit scope vs. mark

  • RI: Repeated Invocation

    • Same command 24x


Learning effects l.jpg
Learning effects devices

  • Timeout plateaus quickly

  • Button gets worse

  • Pigtail, multi-strokefollow similar trend

  • Pigtail problems:

    • Fail to accept

    • Heading in wrong direction

    • Curl too much  choose wrong direction


Pigtail self revelation technique l.jpg
Pigtail self-revelation technique devices

  • Timeout easy to learn but tedious

    • Pause Scriboli shows how to draw pigtail

    • Allows experienced users to shortcut timeout

    • Helps novices learn the pigtail (?)


Ongoing phrasing together multiple gesture strokes l.jpg
ONGOING: Phrasing together multiple gesture strokes devices

  • Automatic means: hard to group arbitrary strokes across various application domains

  • Time delay: “Time outs are lame”

  • Scriboli uses muscular tension on Gesture button (lit.: may help reduce mode errors)


Separating the pig from the tail l.jpg
Separating the pig from the tail devices

  • “+” symbol indicatesgesture is extension ofcurrent command phrase

  • Solves “screen edge” problem very nicely



Grammar for pen input61 l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separate words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope (lasso, line, …)

Marking menu

Crossing manip. phase

Grammar for pen input

  • Scriboli proposes fundamental building blocks

  • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases


Crossing scope line l.jpg
Crossing Scope devices(line)

  • Draw stroke through object

  • Crossing can have different semantics than lassoing

    • Brings up crossing menu

  • e.g. the crossing line specifies right edge of icon for alignment





Crossing scope design properties l.jpg
Crossing Scope devicesdesign properties

  • For cmds that require edge and/or axis

  • Only possible to invoke withrequisite spatial information

  • Limits possible errors of syntax

  • Tremendous economy of design

    • e.g. 1 Align cmd instead of 6












Grammar for pen input77 l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separate words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope (lasso, line, …)

Marking menu

Crossing manip. phase

Grammar for pen input

  • Scriboli proposes fundamental building blocks

  • Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases


Future work l.jpg

Punct. devices

space

capitals

period

comma

object

verb

indirect obj.

Written text

separates words

sentence start

sentence end

delimits clause

“The cat

sat

on the mat”

Scriboli Equivalent

Pen up/pen down

Gesture button down

Gesture button up

Pigtail (delimiter)

Scope (lasso, line, …)

Marking menu

Crossing manip. phase

Future work

  • Are there building blocks missing?

  • What are alternative designs for each part?

  • What other phrases can be envisioned?


Questions l.jpg
Questions? devices


ad