Controlled user studies
1 / 19

Controlled User studies - PowerPoint PPT Presentation

  • Uploaded on

Controlled User studies. HCI - 4163/6610 Winter 2013. Usability Experiments. Predict the relationship between two or more variables. Independent variable is manipulated by the researcher. Dependent variable depends on the independent variable.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Controlled User studies' - koko

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Controlled user studies

Controlled User studies

HCI - 4163/6610

Winter 2013

Usability experiments
Usability Experiments

  • Predict the relationship between two or more variables.

  • Independent variable is manipulated by the researcher.

  • Dependent variable depends on the independent variable.

  • Typical experimental designs have one or two independent variable.

  • Validated statistically & replicable.


True experiment
True Experiment

  • Experimental control

  • Control as many potential threats to validity as possible

  • Random assignment of participants/data to conditions

  • Could be within-subjects or between-subjects


  • True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects

    • Control over the who, what, when, where, how

  • Control of the who => random assignment to conditions

    • Only by chance can other variables be confounded with IV

  • Control of the what/when/where/how => control over the way the experiment is conducted

Quasi experiment

  • When you can’t achieve complete control

    • Lack of complete control over conditions

    • Subjects for different conditions come from potentially non-random pre-existing groups (smokers vs nonsmokers)

It s a matter of control
It’s a matter of control

True Experiment

Quasi Experiment

  • Random assignment of subjects to condition

  • Manipulate the IV

  • Control allows ruling out of alternative hypotheses

  • Selection of subjects for the conditions

  • Observe categories of subjects

    • If the subject variable is the IV, it’s a quasi experiment

  • Don’t know whether differences are caused by the IV or differences in the subjects

Other features
Other features

  • In some instances cannot completely control the what, when, where, and how

    • Need to collect data at a certain time or not at all

    • Practical limitations to data collection, experimental protocol


  • Internal validity is reduced due to the presence of controlled/confounded variables

    • But not necessarily invalid

  • It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences

    • Need to convince self and audience of the validity

External validity
External validity

  • If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting

  • Often comes down to what is most important for the research question

    • Control or ecological validity?


  • Factors: Independent Variables (Ivs) of an experiment

  • Level: particular value of an IV

  • Condition: a group or treatment (technique)

    • e.g., Condition 1: old system, Condition 2: new system

  • Treatment: a condition of an experiment

  • Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

Factors to treatments
Factors to Treatments

  • At least 1 Factor (IV) has to vary to have an experiment

    • Effect of screen size and input technique on performance (speed, accuracy)

  • An IV must always have at least 2 levels

  • Condition refers to a particular way that subjects are treated

    • Between subject: experimental conditions are the same as the groups

    • Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment)

Good experimental design
Good Experimental Design

  • Two-Group, Post-Test Design

  • Two conditions

  • Two groups:

    • Between subjects: random allocation

  • Treatment

  • Post-test: measure the DV

  • What’s really important?

Experimental designs
Experimental designs

  • Between subjects: Different participants - single group of participants is allocated randomly to the experimental conditions.

  • Within subjects: Same participants - all participants appear in both conditions.

  • Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.


Within subjects

  • Similar to the one-group pre-test-post-test design

  • It solves the individual differences issues

  • But raises other problems:

    • Need to look at the impact of experiencing the two conditions

    • Will they get tired? Gain practice? Learn what is expected?

  • Need to control for order and sequence effects?

Order effects
Order Effects

  • Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?)

  • Arises from warm-up, learning, fatigue, etc.

  • Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

Sequence effects
Sequence effects

  • Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2)

  • Effects viewed may not be main effects of the IV, but interaction effects

  • Can be controlled by arranging each condition to follow every other condition equally often


  • Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders

  • Self-directed learning: investigate the different counterbalancing methods

    • Randomization

    • Block Randomization

    • Reverse counter-balancing

    • Latin squares and Greco squares (when you can’t fully counterbalance)


Between within matched participant design
Between, within, matched participant design


Key points 1
Key points 1

  • Usability testing is done in controlled conditions.

  • Usability testing is an adapted form of experimentation.

  • Experiments aim to test hypotheses by manipulating certain variables while keeping others constant.

  • The experimenter controls the independent variable(s) but not the dependent variable(s).