Sketching interfaces for specification
1 / 40

Sketching Interfaces for Specification - PowerPoint PPT Presentation

  • Updated On :

Sketching Interfaces for Specification Readings James A. Landay, Brad A. Myers, “Sketching Interfaces: Toward More Human Interface Design”, IEEE Computer , v34, n3, pp. 56-64, 2001.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Sketching Interfaces for Specification' - Audrey

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Readings l.jpg

James A. Landay, Brad A. Myers, “Sketching Interfaces: Toward More Human Interface Design”, IEEE Computer, v34, n3, pp. 56-64, 2001.

Bailey, B.P., J.A. Konstan, and J.V. Carlis, "DEMAIS: designing multimedia applications with interactive storyboards", Proceedings of the International Conference on Multimedia, 2001, Ottawa, pp. 241-250.

Supporting early design l.jpg
Supporting Early Design

  • Key tasks being supported

    • Design space exploration

      • Medium for expression and “visual thinking”

        • Increase concreteness

        • Providing inspiration and points of departure

      • Understanding the implications of design choices

    • Communications

      • Vehicle for talking about design options with others

Sketching is good l.jpg
Sketching is Good

  • Almost everyone starts in early stage with very rough pencil and paper sketches

    • Rough ideas quickly

    • Uncertainty

      • Uncertainty and ambiguity is important to have at this stage (important to creativity)

      • Can’t have the details before you have done the design

    • Fluidity is critical

      • Investment must be very low

      • Changes must be very easy

      • Must stay “light on your feet” and not get solidified into small range of alternatives too early

Need ability to test operate as early as possible l.jpg
Need ability to test / operate as early as possible

  • But also need to be able to evaluate the real meaning / consequence of things

    • Must make them “operable” or executable at some level to get a real understanding

    • Big win

  • Herein lies the hard problem: Ambiguity and uncertainty are very antithetical to being able to “execute” something

Getting past the divide l.jpg
Getting past the divide

  • Being able to “execute” something which is ambiguous and only partially defined

    • Need to infer intent

    • Need sensible interpretation of all partial specifications

    • Hard: probably need to limit domain

  • Also need graceful path from early/ambiguous to later/precise (“real”)

    • Would rather not be forced to “throw it away” after a certain point

Slide7 l.jpg

  • Sketch

    • On digitizing tablet rather than paper

      • Interactive rather than static setting

      • Turn sketches into something that has enough structure and meaning that rich computational things can be done with it

      • Try to get fluidity benefits of pen but add a lot more

  • Storyboard

    • Used as central mechanism for specifying behavior

    • Limited, but well targeted (and already familiar to many)

  • Transition to real widgets

    • Gives (some) “upward path” for results

Silk modes l.jpg
SILK modes

  • Sketch

  • Annotate

  • Decorate

  • Run

  • Also use of pen mode

    • Special button on pen for command gestures

  • Modes help a lot with recognition

Silk interfaces l.jpg
SILK interfaces

  • Sketches

    • Window for sketching (specifying and placing) interactive components (widgets)

    • Screen at a time or dialog box at a time

      • Sort of assuming static layout and somewhat moded style for result

  • Storyboard

    • Window which contains collected copies of sketched screens

    • Place to specify behavior

Sketching widgets l.jpg
Sketching widgets

  • Attempt to recognize meaningful objects

    • Key to executability

  • Make recognized sketches into operating widgets

    • Still appear as sketches

  • Later transform into “real” widgets

Widget set l.jpg
Widget set

  • Seven widgets in fixed set

    • Menu bar

    • Scrolling window

    • Palette

    • Button

    • Radio button

    • Check box

    • Text Field

  • Plus rows and columns (“panels”)

Fixed set shows up as a significant limitation in user tests

But supporting an arbitrary set introduces big programming issues

Widgets recognized from primitive components l.jpg
Widgets recognized from primitive components

  • Four primitive gestures recognized

    • Rectangle

    • Squiggly Line

    • Straight Line

    • Ellipse

  • All single stroke

    • May require slight adjustment by user

    • Makes life a lot easier for recognizer

      • No segmentation

  • Rubine recognizer

Recognition l.jpg

  • Rubine recognizer

    • Single stroke

    • Feature based

      • E.g., number of inflection points, angles, pt-pt distances, etc…

    • Not person independent

      • Trained w/ 15-20 examples

      • Also feedback during use (via corrections made)

Recognition15 l.jpg

  • Steps

    • Individual component recognition

    • Spatial relationship analysis

    • Widget inference

    • Look for higher level groupings

    • Feedback

    • Possible user correction

Recognition steps l.jpg
Recognition steps

  • Spatial relationship analysis

    • Relationships recognized

      • Contains / is contained by

      • Near

        • Left, right, above, below

      • Vertical or horizontal sequence

        • (of the same type)

Recognition steps17 l.jpg
Recognition Steps

  • Rule based widget inference

    <pattern> <widget selection, confidence, code>

    • Pattern

      • Look for certain configurations of components

        • Rectangle contains rectangle and outer has very vertical aspect ratio  scroll bar

    • Body

      • Compute confidence from details of matched parts

      • Provide code for operating as a widget

Recognition steps18 l.jpg
Recognition Steps

  • Look for “panels”

    • Horizontal or vertical groupings of widgets

      • Proximity

      • Alignment

    • Must all be the same type

    • Looks to create new panels or add to existing after each widget recognition

Recognition steps19 l.jpg
Recognition Steps

  • Feedback

    • Recognized widgets turn purple(Not clear what feedback for groupings was)

Usability problem uncovered in evaluation:

Can’t tell which widget was recognized (without looking at control panel), but if you don’t get it right, correct panels don’t work

Recognition steps20 l.jpg
Recognition Steps

  • Biggest usability problem will turn out to be recognition errors

  • User correction

    • Command gesture for “next best guess”

      • Users can also give “hints” via selections

    • Explicit change in control panel

    • If no widget found control panel has “New guess” button to tell system to “look harder”

Also have recognition of command gestures l.jpg








Also have recognition of command gestures

  • Recall: special “command” button on pen

    • Very common approach

  • All structured as a single stroke

Specifying behavior via storyboards l.jpg
Specifying behavior via storyboards

  • Can place individual screens (collections of ink, widgets, etc.) into storyboard view

    • Copy/paste then edit

  • All behavior specified as navigation in story board

    • Specified by drawing a line (in storyboard editor) from either widget or background to a new scene.

    • In run-mode click on that object means “goto …”

Annotation and decoration l.jpg
Annotation and decoration

  • Decoration mode

    • Basically turns off the recognizer

    • All strokes just in as unrecognized ink

  • Annotation mode

    • Attaches ink and text to whole screen or individual widgets

    • Annotations can be turned on and off

Run mode l.jpg
Run mode

  • Interface (with same sketched appearance) runs

    • Scroll bars slide, etc.

    • Buttons, etc. cause movements in storyboard

      • May not look that way to the user (e.g., single change between boards)

  • Also supplies some debug help

    • Storyboard highlights current and object which caused last last transition

Transition to real interfaces l.jpg
Transition to “real” interfaces

  • Can ask for sketched widgets to be turned into “real” widgets

    • VB or Garnet

  • Still need to add:

    • Precise alignments

    • Color

    • Text labels

    • Actual icons images, etc

    • Real action code!

Experience usability l.jpg
Experience & usability

  • Overall seems very positive

  • Biggest issue: recognition errors

    Recognition rates:

    Edit gestures: 89%

    Primitives: 93%

    Widgets: 69%  not going to work

    End analysis:

    • Recognition probably worse than a more explicit strategy

Experience usability28 l.jpg
Experience & usability

  • Second big issue: fixed widget set

    • Can do some things, but support falls way off for “non-standard” interactions

    • A lot of effort to work around with inadequate tools

    • Hard problem because taking on large part of “the programming problem” may hurt simpler and more common aspect

    • Interesting research issues here

Demais system l.jpg
Demais System

  • New (but related) domain: Multimedia presentations

    • Time based media (video and audio)

    • Timing and synchronization behavior now the tricky part

      • Understanding design choices requires understanding how the detailed timing and sync will play out

  • Contribution is in specification of behavior

    • A more complex programming task (but still limited)

Demais interface components l.jpg
Demais interface components

  • Again, storyboard based

    • Single screen (layout) editor

    • Narration editor

      • Deals with audio narration

    • Storyboard editor

    • Also “multi-view” editor

      • Collections of screens, storyboards, etc. that are useful to pull up and manipulate together

      • Sort of a “scrapbook” manager

Entities system deals with in a layout screen l.jpg
Entities system deals with in a layout screen

  • Plain text objects

    • Interpreted and uninterpreted

    • (Can do synch spec in scripting language)

  • Ink strokes

    • uninterpreted

  • Recognized objects

  • Behavioral ink strokes

  • Visual language icons

  • Annotations

Recognition32 l.jpg

  • Very simple for sketched object recognition

    • Only recognizes two things (Rubine algorithm again)

      • Rectangle

        • Indicates “content item” (still or video)

        • Tap to get file browser to load actual or simulated content

      • Behavior stroke

        • Single line from something to something

    • Otherwise treat as uninterpreted ink

  • Text recognition

    • Tries to parse as script text

    • If fails, uninterpreted

Narration editor l.jpg
Narration editor

Can insert “synchronization markers” in text

  • Points to synchronize other stuff against

  • System does text-to-speech

  • Also supports recorded speech but not clear how sync points are specified

    • Probably like video (position of slider when behavior attached), but not explicitly stated

  • Narration editor34 l.jpg
    Narration editor

    • To synchronize,

      • e.g., image appearance

        with a point in the narration sketch a (behavior) stroke from the sync point to another storyboard element

    Visual language for behaviors l.jpg
    Visual language for behaviors

    • Behavior strokes are between objects

      • Creates implicit parameter and subject from source and destination of the stroke

    • Strokes annotated to specify their actual behavior

      • Triggering event (from source)

      • Action to perform (at destination)

    Behavior icons events l.jpg
    Behavior iconsEvents

    • “T”s replaced by numbers

      • For sync: use current time of object

      • For elapsed: user prompted

    Can replace defaults l.jpg
    Can replace defaults

    • Edit by tap-and-drop from palette

      • Better than drag-and-drop with pen

    • Can also use text annotations in scripting language for specifying behaviors

      • English like (details not discussed here)

      • Infers referents (“this”)

      • Doesn’t seem to fit well