Interaction modalities

1. Interaction modalities Command languages, direct manipulation, and WIMP

2. What’s wrong with this? 0. Borrow a book from the library 1. go to the library 2. log in to library catalogue 2.1 access the search screen 2.2 enter search criteria 2.3 identify required book 2.4 note location 3. go to correct shelf and retrieve book 3.1 use library map to find shelf 4. take book to checkout counter

3. Now what’s wrong? 0. In order to borrow a book from the library 1. go to the library 1.1 enter library 1.2 go to computer 1.3 log into catalogue 2. find book 2.1 access library catalogue 2.2 access the search screen 2.3 enter search criteria 2.4 identify required book 2.5 note location 3. go to correct shelf and retrieve book 4. check out book

4. Don’t forget your plans! 0. In order to borrow a book from the library 1. go to the library 2. find the required book 2.1 access library catalogue 2.2 access the search screen 2.3 enter search criteria 2.4 identify required book 2.5 note location 3. go to correct shelf and retrieve book 4. check out book Plan 0: do 1, 2, 3; repeat 2, 3, if book not found; then do 4 Plan 2: do 2.1, 2.2; repeat 2.3 until found; 2.4, 2.5

5. Interface types 1980s interfaces Command line WIMP/GUI 1990s interfaces Advanced graphical (multimedia, virtual reality, information visualization) Web Speech (voice) Pen, gesture, and touch Appliance 2000s interfaces Mobile Multimodal Shareable Tangible Augmented and mixed reality Wearable Robotic

6. Interaction Styles • 1. Command languages • 2. WIMP - Window, Icon, Menu, Pointer • 3. Direct manipulation • 4. Speech/Natural language • 5. Pen & Touch

7. General Issues in Interaction Style • Who is in control - user or computer • Initial training required • Learning time to become proficient • Special skills • Speed of use • Generality/flexibility • Power • Screen space required • Computational resources required • Match to the user’s activity

8. Command Languages • Earliest UI interaction paradigm • Examples: MS-DOS shell, UNIX, Linux • Little or nothing is visible so… • Work primarily by recall, not recognition • Heavy memory load • Poor choice for novices but...

9. CL Advantages • Advantages for experts • Speed, conciseness % ls (hard to beat) • Can express actions beyond a limited set • Flags, piping one command to another • Repetition, extensibility • Scripting, macros • Easier implementation, less overhead • Power • Abstraction, wild cards

10. CL Dangers • With added power, comes added responsibility and danger • UNIX % rm -r * • Deletes every file that you have, and you can’t get them back

11. CL Design Goals • Consistency • Have options and arguments expressed the same way everywhere • Good naming and abbreviations • UNIX fails here because commands were developed by lots of different people at different organizations • No guidelines provided

12. Names and Abbreviations • Specificity versus Generality • General words • More familiar, easier to accept • Specific (typically better) • More descriptive, meaningful, distinctive • (Nonsense does surprisingly well in small set) • Abbrevs. allow for faster actions • Expert performance begins to be dominated by motor times such as # of keystrokes • Not good idea for novices • (Allow but don’t require)

13. General Issues - CL • Initial training required • Learning time to become proficient • Special skills • Speed of use • Generality/flexibility • Power • Screen space required • Computational resources required • Match to the user’s activity

14. Interaction Styles • 1. Command language • 2. WIMP • 3. Direct Manipulation • 4. Speech/Natural language • 5. Gesture, pen, VR

15. WIMP/GUI interfaces • Xerox Star first WIMP -> rise to GUIs • Windows • could be scrolled, stretched, overlapped, opened, closed, and moved around the screen using the mouse • Icons • represented applications, objects, commands, and tools that were opened when clicked on • Menus • offering lists of options that could be scrolled through and selected • Pointing device • a mouse controlling the cursor as a point of entry to the windows, menus, and icons on the screen

16. Windows • Windows were invented to overcome physical constraints of a computer display • Overlapping windows make more efficient use of screen real estate • Can allow dragging and dropping between them • But can become cluttered or overwhelming • Multiple windows can make it difficult to find desired one, so techniques used • Listing, iconising, shrinking

17. Questions • How many windows do you generally have open at once? • How do you arrange them? • How do you switch between them?

18. Menus • Advantages: • 1 keystroke or mouse operation vs. many • No memorization of commands • Limited input set • Disadvantages: • Less direct user control - have to find correct menu / menu item • Not so readily extensible • Slower than keyboarding for experienced users, at least without accelerators

19. Menu Items • Various types: • Flat • Cascading • Pop-up (contextual) • Organization strategies • Create groups of logically similar items • Cover all possibilities • Ensure that items are non-overlapping • Keep wording concise, understandable

20. A Good Example • Logical grouping • Visual separation of groups • Disabled items “grayed out” • Shortcuts shown • … indicates leads to dialogue

21. Icons • Icons are assumed to be easier to learn and remember than commands • Can be designed to be compact and variably positioned on a screen • Issues: • Consistency • Legibility • Convey meaning

22. Simple icons

23. Simple icons

24. Newer icons

25. Pointers • Choosing commands from menus and icons • Allows users to specify location parameter of a command • Edit HERE, Put this file HERE • Also displays system state info: • Tracking • Busy • Hints • Modes (text cursor vs. selection pointer)

26. General Issues - WIMP • Who is in control - user or computer • Initial training required • Learning time to become proficient • Special skills • Speed of use • Generality/flexibility • Power • Screen space required • Computational resources required • Match to the user’s activity

27. Direct Manipulation Essence • Representation of reality that can be manipulated • The user is able to apply intellect directly to the task • The tool itself seems to disappear

28. Direct Manipulation Definition: • 1) Continuous visibility of the objects and actions of interest • 2) Rapid, reversible, incremental actions whose effect is immediately noticeable • 3) Replacement of command language syntax by direct manipulation of object of interest (physical actions, buttons, etc.) Shneiderman ‘82

29. Direct Manipulation • Examples • WYSIWYG editors and word processors • VISICALC - 1st electronic spreadsheet • CAD • Desktop metaphor • Video games

30. Example: maps

31. Execution Goals System Evaluation More Psychological View • What is directness? (not always done well) • Related to two things: • Distance • Engagement • Unobtrusive and responsive Hutchins, Hollan, Norman ‘86

32. Example: Homefinder

33. DM Advantages • Easier to learn & remember, particularly for novices • Flexible, easily reversible actions helps reduce anxiety in users • Provides context & instant visual feedback so user can tell if objectives are being achieved • Exploits human use of visual spatial cues • Limits types of errors that can be made

34. DM Problems • Screen space intensive (info not all that dense) • Need to learn meaning of components of visual representation • Visual representation may be misleading • Mouse ops may be slower than typing • Not self-explanatory (no prompts) • Not everything can be represented visually

35. DM Problems • Not good at • Repetition • History keeping (harder) • Certain tasks (Change all italics to bold) • Abstract elements (variables) • Macros harder

36. General Issues - DM • Initial training required • Learning time to become proficient • Special skills • Speed of use • Generality/flexibility • Power • Screen space required • Computational resources required • Match to the user’s activity

37. Interaction styles: next time • 1. Command language • 2. WIMP • 3. Direct Manipulation • 4. Speech/Natural language • 5. Touch and Pen