Making sense out of recorded user system interaction
Download
1 / 48

Making sense out of recorded user-system interaction - PowerPoint PPT Presentation


  • 72 Views
  • Uploaded on

Making sense out of recorded user-system interaction. Dr Willem-Paul Brinkman Lecturer Department of Information Systems and Computing Brunel University ([email protected]). Topics. VIVID Research Centre Motivation - Component-Based Software Engineering

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Making sense out of recorded user-system interaction' - maxime


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Making sense out of recorded user system interaction

Making sense out of recorded user-system interaction

Dr Willem-Paul Brinkman

Lecturer

Department of Information Systems and Computing

Brunel University

([email protected])


Topics
Topics

  • VIVID Research Centre

  • Motivation - Component-Based Software Engineering

  • Experiment 1: Searching for a component-specific measure

  • Experiment 2: Validating a component-specific measure

  • New and future research


Vivid research centre
VIVID Research Centre

  • Based in the Department of Information Systems and Computing, Brunel University (London)

  • Original focus on visualisation, but now also includes:

  • - Mobile technology

  • Design for diverse user groups

  • - Novel input/output devices

  • 11 academics member of staff, 13 PhD Students

disc.brunel.ac.uk/research/vivid/index.htm


Motivation
Motivation

Studying the usability of a system

Work conducted together with Reinder Haakma (Philips), Don Bouwhuis (Eindhoven University of Technology)


Motivation1

ExternalComparison relating difference in usability to differences in the systems

Motivation

InternalComparisontrying to link usability problems with parts of the systems


Component based software engineering
Component-Based Software Engineering

  • Multiple versions testing paradigm

  • Single version testing paradigm

Manage

Create

Support

Re-use

Re-use


Motivation2
Motivation

  • PROBLEM

  • Only empirical analysis of the overall system such as (task time, keystrokes, questionnaires etc) - not powerful

  • Usability tests, heuristic evaluations, cognitive walkthroughs where experts problems – unreliable

  • SOLUTION

  • Component-Specific usability measures: more powerful and reliable


Searching for a component specific measure
Searching for a component-specific measure

  • Questions

  • What is a component?

  • What interaction data should be recorded?

  • How do we link interaction data with the usability of a component?



Interaction layers

Editor

Processor

38

15 + 23

15 +

15

38

15 + 23

15 +

15

Interaction layers

Control

results

01111

10111

Add

15 + 23 =

100110

38

Control

equation

15

=

23

+

User

Calculator


Experiment 1 fictitious interface
Experiment 1 – Fictitious Interface

User Task: Rotate the Trumpet


Experiment 1 architecture

Rotator

Rotate(x)

Map

X

Selector

Rotate

Change

Buttons

Bike

Aeroplane

Experiment 1 - Architecture

Other symbols


Experiment 1 architecture1

High

Rotator

Rotate(x)

Map

X

Selector

Rotate

Change

Buttons

Low

Bike

Aeroplane

Experiment 1 - Architecture

Measures

Task time

#Rotate(T0),

#Rotate(T-1),

#Rotate(T-2)

#change, #rotate

#bike,#aeroplane, #other

#clicks

Other symbols


Experiment 1 training
Experiment 1 - Training


Experiment 1 test procedure
Experiment 1 : Test Procedure

80 participants, all students of Eindhoven University of Technology

8 different trainings

After training participants were asked to rotate, as fast as possible, a specific music instrument

User interaction with the system was recorded in log file

Once a task was complete the recording stops


Experiment 1 low level effect of selector training
Experiment 1 - Low-level Effect of Selector training

Number messages

Clicks on


Experiment 1 high level effect rotator training
Experiment 1 - High-level Effect Rotator Training

Number messages

#Rotate (X)


Experiment 1 control loop

Reference value

Evaluation

User message

Feedback

Component

Experiment 1 – Control Loop

Reliability: how do we link interaction data with the usability of a component?

Each message is a cycle of the control loop  Number of messages presents the user’s effort to control the component

User

System


Experiment 1 conclusion
Experiment 1 - Conclusion

  • What is a component?

    • An interaction component is a unit within a device that directly or indirectly receives signals from the user. These signals enable the user to change the physical state of the interaction component

  • What interaction data should be recorded?

    • Message exchange between the interaction components


Experiment 2 validation
Experiment 2 : Validation

80 users

8 mobile telephones

3 components were manipulated according to Cognitive Complexity Theory (Kieras & Polson, 1985)

  • Function Selector

  • Keypad

  • Short Text Messages


Architecture mobile telephone
Architecture Mobile telephone

Send Text

Message

Function

Selector

Keypad


Experiment 2 function selector
Experiment 2 – Function Selector

  • Versions:

    • Broad/shallow

    • Narrow/deep


Experiment 2 keypad

Repeated-Key Method

“L”

Modified-Model-Position method

“J”

Experiment 2 – Keypad

Versions


Experiment 2 send text message
Experiment 2 – Send Text Message

Simple

Versions

Complex


Statistical tests
Statistical Tests

p-value: probability of making type I, or , error, wrongly rejecting the hypothesis that underlying distribution is same.


Results function selector
Results – Function Selector

Results of two multivariate analyses and related univariate analyses of variance with the version of the Function Selector as independent between-subjects variable.


Results keypad
Results – Keypad

Results of multivariate and related univariate analyses of variance with the version of the Keypad as independent between-subjects variable.


Results send text message
Results – Send Text Message

Results of two multivariate analyses and related univariate analyses of variance with the version of the STM component as independent between-subjects variable



Results
Results

Average probability that a measure finds a significant (α = 0.05) effect for the usability difference between the two versions of FS, STM, or the Keypad components


Component based software engineering1
Component-Based Software Engineering

  • Multiple versions testing paradigm

  • Single version testing paradigm

Manage

Create

Support

Re-use

Re-use


Testing different components
Testing Different Components

  • Component specific objective performance measure:

    • Messages received + Weight factor

      A common currency

    • Compare with ideal user

      A common point of reference

  • Usability of individual components in a single device can be compared with each other and prioritized on potential improvements


Assigning weight factors to represent the user s effort in the case of ideal user

Right Mouse

Button Menu

Properties

Assigning weight factors to represent the user’s effort in the case of ideal user

{7}

Set <Fill colour red, no border>

{2}

Call <>

{1}

Click <left on Fill tab>

{1}

Click <left on on colour red>

{1}

Click <left on Outline tab>

{1}

Click <left No Line button>

{1}

Click <right>

{1}

Click <left no Ok button>

{1}

Click <left on Properties option>


Total effort value

Properties

Right Mouse

Button Menu

Total effort value

  • Total effort =  MRi.W

  • MRi.W : Message received. Weight factor

{2}

Call <>

{1}

Click <left on Fill tab>

{1}

Click <left on on colour red>

{1}

Click <left on Outline tab>

{1}

Click <left No Line button>

{1}

Click <right>

{1}

Click <left no Ok button>

{1}

Click <left on Properties option>

5

+

2 = 7

2


Assigning weight factors in case of real user

Visual

Drawing

Objects

Properties

Right Mouse

Button Menu

Assigning weight factors in case of real user

Correction for inefficiency of higher and lower components


Assigning weight factors in case of real user1

Visual

Drawing

Objects

Properties

Right Mouse

Button Menu

Assigning weight factors in case of real user

Inefficiency of lower level components: need more messages to pass on a message upwards than ideally required

Assign weight factors as if lower components operate optimal


Assigning weight factors in case of real user2

Visual

Drawing

Objects

Properties

Right Mouse

Button Menu

 MRi.W

#MSU ideal

UE =

#MSU real

Assigning weight factors in case of real user

Inefficiency of higher level components: more messages are requested than ideally required

  • UE : User effort

  • MRi.W : Message received. Weight factor

  • #MSUreal :Number of messages sent upward by real user

  • #MSUideal :Number of messages sent upward by ideal user


Ideal user versus real user

The total effort an ideal user would make

The total effort a real user made

The extra effort a real user made

Prioritize

Ideal User versus Real User

  • Extra User Effort = User Effort - Total effort

Calculate for each component:


Experiment 2 single version
Experiment 2 - Single version

40 users

4 mobile telephones

2 components were manipulated (Keypad only Repeated-Key Method)

  • Function Selector

  • Short Text Messages


Results1
Results

Extra User Effort

Mobile phones


Results2
Results

Partial correlation between extra user effort regarding the two components and other usability measures

*p. < .05. **p. < .01.


Topics1
Topics

  • VIVID Research Centre

  • Motivation - Component-Based Software Engineering

  • Experiment 1: Searching for a component-specific measure

  • Experiment 2: Validating a component-specific measure

  • New and future research

    • Extending the analysis outside the lab

    • Extending the analysis beyond only usability issues


New projects field usability
New Projects - Field usability

  • CD player, which 10 users will use at home

  • Record interaction: online assignment of weigh factors, both optimal and real user, to messages

  • Correlated interaction data with other data (questionnaire, dairy, interview)

(Pui-Fong Man)


New projects proskin
New Projects - PROSKIN

User profiling for skinnable domestic technology

  • Exciting Interface designed for the average user. However, the average user does not exist.

  • Developing skins for specific user groups could be a way forward

  • Question:

    • How to identify user groups?

    • What do user groups want?

Work conducted together with Nick Fine


New projects proskin1
New Projects - PROSKIN

User profiling for skinnable domestic technology

  • Possible solution

  • Recording online interaction, Identifying user groups, Developing skins for these user groups

  • Question

  • How to establish user groups that are relevant for designer?

  • This time, how to make sense of the interaction data beyond usability?

Work conducted together with Nick Fine


New projects proskin2

Online Validation

User metrics

Interaction data

User groups based on interaction data

Design of Skins

New Projects - PROSKIN

Approach


Conclusions and final remarks
Conclusions and Final Remarks

  • Interaction data can be used to study the usability of interaction components

    • External Comparison between different versions: More Powerful

    • Internal Comparison:prioritized on potential improvements

  • Future questions

    • Usability analysis of everyday life interaction

    • Establishing new paradigms to understand interaction data beyond usability issues


  • Questions

    Questions

    Thank you for your attention


    ad