Making sense out of recorded user-system interaction

1. Making sense out of recorded user-system interaction Dr Willem-Paul Brinkman Lecturer Department of Information Systems and Computing Brunel University (willem.brinkman@brunel.ac.uk)

2. Topics • VIVID Research Centre • Motivation - Component-Based Software Engineering • Experiment 1: Searching for a component-specific measure • Experiment 2: Validating a component-specific measure • New and future research

3. VIVID Research Centre • Based in the Department of Information Systems and Computing, Brunel University (London) • Original focus on visualisation, but now also includes: • - Mobile technology • Design for diverse user groups • - Novel input/output devices • 11 academics member of staff, 13 PhD Students disc.brunel.ac.uk/research/vivid/index.htm

4. Motivation Studying the usability of a system Work conducted together with Reinder Haakma (Philips), Don Bouwhuis (Eindhoven University of Technology)

5. ExternalComparison relating difference in usability to differences in the systems Motivation InternalComparisontrying to link usability problems with parts of the systems

6. Component-Based Software Engineering • Multiple versions testing paradigm • Single version testing paradigm Manage Create Support Re-use Re-use

7. Motivation • PROBLEM • Only empirical analysis of the overall system such as (task time, keystrokes, questionnaires etc) - not powerful • Usability tests, heuristic evaluations, cognitive walkthroughs where experts problems – unreliable • SOLUTION • Component-Specific usability measures: more powerful and reliable

8. Searching for a component-specific measure • Questions • What is a component? • What interaction data should be recorded? • How do we link interaction data with the usability of a component?

9. Layered Protocol Theory(Taylor, 1988)

10. Editor Processor 38 15 + 23 15 + 15 38 15 + 23 15 + 15 Interaction layers Control results 01111 10111 Add 15 + 23 = 100110 38 Control equation 15 = 23 + User Calculator

11. Experiment 1 – Fictitious Interface User Task: Rotate the Trumpet

12. Rotator Rotate(x) Map X Selector Rotate Change Buttons Bike Aeroplane Experiment 1 - Architecture Other symbols

13. High Rotator Rotate(x) Map X Selector Rotate Change Buttons Low Bike Aeroplane Experiment 1 - Architecture Measures Task time #Rotate(T0), #Rotate(T-1), #Rotate(T-2) #change, #rotate #bike,#aeroplane, #other #clicks Other symbols

14. Experiment 1 - Training

15. Experiment 1 : Test Procedure 80 participants, all students of Eindhoven University of Technology 8 different trainings After training participants were asked to rotate, as fast as possible, a specific music instrument User interaction with the system was recorded in log file Once a task was complete the recording stops

16. Experiment 1 - Low-level Effect of Selector training Number messages Clicks on

17. Experiment 1 - High-level Effect Rotator Training Number messages #Rotate (X)

18. Reference value Evaluation User message Feedback Component Experiment 1 – Control Loop Reliability: how do we link interaction data with the usability of a component? Each message is a cycle of the control loop  Number of messages presents the user’s effort to control the component User System

19. Experiment 1 - Conclusion • What is a component? • An interaction component is a unit within a device that directly or indirectly receives signals from the user. These signals enable the user to change the physical state of the interaction component • What interaction data should be recorded? • Message exchange between the interaction components

20. Experiment 2 : Validation 80 users 8 mobile telephones 3 components were manipulated according to Cognitive Complexity Theory (Kieras & Polson, 1985) • Function Selector • Keypad • Short Text Messages

21. Architecture Mobile telephone Send Text Message Function Selector Keypad

22. Experiment 2 – Function Selector • Versions: • Broad/shallow • Narrow/deep

23. Repeated-Key Method “L” Modified-Model-Position method “J” Experiment 2 – Keypad Versions

24. Experiment 2 – Send Text Message Simple Versions Complex

25. Statistical Tests p-value: probability of making type I, or , error, wrongly rejecting the hypothesis that underlying distribution is same.

26. Results – Function Selector Results of two multivariate analyses and related univariate analyses of variance with the version of the Function Selector as independent between-subjects variable.

27. Results – Keypad Results of multivariate and related univariate analyses of variance with the version of the Keypad as independent between-subjects variable.

28. Results – Send Text Message Results of two multivariate analyses and related univariate analyses of variance with the version of the STM component as independent between-subjects variable

29. Power of number of messages as a usability measure Statistical Power: 1 - β

30. Results Average probability that a measure finds a significant (α = 0.05) effect for the usability difference between the two versions of FS, STM, or the Keypad components

31. Component-Based Software Engineering • Multiple versions testing paradigm • Single version testing paradigm Manage Create Support Re-use Re-use

32. Testing Different Components • Component specific objective performance measure: • Messages received + Weight factor A common currency • Compare with ideal user A common point of reference • Usability of individual components in a single device can be compared with each other and prioritized on potential improvements

33. Right Mouse Button Menu Properties Assigning weight factors to represent the user’s effort in the case of ideal user {7} Set <Fill colour red, no border> {2} Call <> {1} Click <left on Fill tab> {1} Click <left on on colour red> {1} Click <left on Outline tab> {1} Click <left No Line button> {1} Click <right> {1} Click <left no Ok button> {1} Click <left on Properties option>

34. Properties Right Mouse Button Menu Total effort value • Total effort =  MRi.W • MRi.W : Message received. Weight factor {2} Call <> {1} Click <left on Fill tab> {1} Click <left on on colour red> {1} Click <left on Outline tab> {1} Click <left No Line button> {1} Click <right> {1} Click <left no Ok button> {1} Click <left on Properties option> 5 + 2 = 7 2

35. Visual Drawing Objects Properties Right Mouse Button Menu Assigning weight factors in case of real user Correction for inefficiency of higher and lower components

36. Visual Drawing Objects Properties Right Mouse Button Menu Assigning weight factors in case of real user Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required Assign weight factors as if lower components operate optimal

37. Visual Drawing Objects Properties Right Mouse Button Menu  MRi.W #MSU ideal UE = #MSU real Assigning weight factors in case of real user Inefficiency of higher level components: more messages are requested than ideally required • UE : User effort • MRi.W : Message received. Weight factor • #MSUreal :Number of messages sent upward by real user • #MSUideal :Number of messages sent upward by ideal user

38. The total effort an ideal user would make The total effort a real user made The extra effort a real user made Prioritize Ideal User versus Real User • Extra User Effort = User Effort - Total effort Calculate for each component:

39. Experiment 2 - Single version 40 users 4 mobile telephones 2 components were manipulated (Keypad only Repeated-Key Method) • Function Selector • Short Text Messages

40. Results Extra User Effort Mobile phones

41. Results Partial correlation between extra user effort regarding the two components and other usability measures *p. < .05. **p. < .01.

42. Topics • VIVID Research Centre • Motivation - Component-Based Software Engineering • Experiment 1: Searching for a component-specific measure • Experiment 2: Validating a component-specific measure • New and future research • Extending the analysis outside the lab • Extending the analysis beyond only usability issues

43. New Projects - Field usability • CD player, which 10 users will use at home • Record interaction: online assignment of weigh factors, both optimal and real user, to messages • Correlated interaction data with other data (questionnaire, dairy, interview) (Pui-Fong Man)

44. New Projects - PROSKIN User profiling for skinnable domestic technology • Exciting Interface designed for the average user. However, the average user does not exist. • Developing skins for specific user groups could be a way forward • Question: • How to identify user groups? • What do user groups want? Work conducted together with Nick Fine

45. New Projects - PROSKIN User profiling for skinnable domestic technology • Possible solution • Recording online interaction, Identifying user groups, Developing skins for these user groups • Question • How to establish user groups that are relevant for designer? • This time, how to make sense of the interaction data beyond usability? Work conducted together with Nick Fine

46. Online Validation User metrics Interaction data User groups based on interaction data Design of Skins New Projects - PROSKIN Approach

47. Conclusions and Final Remarks • Interaction data can be used to study the usability of interaction components • External Comparison between different versions: More Powerful • Internal Comparison:prioritized on potential improvements • Future questions • Usability analysis of everyday life interaction • Establishing new paradigms to understand interaction data beyond usability issues

48. Questions Thank you for your attention