Controlled user studies
Sponsored Links
This presentation is the property of its rightful owner.
1 / 19

Controlled User studies PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Controlled User studies. HCI - 4163/6610 Winter 2013. Usability Experiments. Predict the relationship between two or more variables. Independent variable is manipulated by the researcher. Dependent variable depends on the independent variable.

Download Presentation

Controlled User studies

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Controlled user studies

Controlled User studies

HCI - 4163/6610

Winter 2013

Usability experiments

Usability Experiments

  • Predict the relationship between two or more variables.

  • Independent variable is manipulated by the researcher.

  • Dependent variable depends on the independent variable.

  • Typical experimental designs have one or two independent variable.

  • Validated statistically & replicable.


True experiment

True Experiment

  • Experimental control

  • Control as many potential threats to validity as possible

  • Random assignment of participants/data to conditions

  • Could be within-subjects or between-subjects



  • True experiment = complete control over the subject assignment to conditions and the presentation of conditions to subjects

    • Control over the who, what, when, where, how

  • Control of the who => random assignment to conditions

    • Only by chance can other variables be confounded with IV

  • Control of the what/when/where/how => control over the way the experiment is conducted

Quasi experiment


  • When you can’t achieve complete control

    • Lack of complete control over conditions

    • Subjects for different conditions come from potentially non-random pre-existing groups (smokers vs nonsmokers)

It s a matter of control

It’s a matter of control

True Experiment

Quasi Experiment

  • Random assignment of subjects to condition

  • Manipulate the IV

  • Control allows ruling out of alternative hypotheses

  • Selection of subjects for the conditions

  • Observe categories of subjects

    • If the subject variable is the IV, it’s a quasi experiment

  • Don’t know whether differences are caused by the IV or differences in the subjects

Other features

Other features

  • In some instances cannot completely control the what, when, where, and how

    • Need to collect data at a certain time or not at all

    • Practical limitations to data collection, experimental protocol



  • Internal validity is reduced due to the presence of controlled/confounded variables

    • But not necessarily invalid

  • It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences

    • Need to convince self and audience of the validity

External validity

External validity

  • If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting

  • Often comes down to what is most important for the research question

    • Control or ecological validity?



  • Factors: Independent Variables (Ivs) of an experiment

  • Level: particular value of an IV

  • Condition: a group or treatment (technique)

    • e.g., Condition 1: old system, Condition 2: new system

  • Treatment: a condition of an experiment

  • Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

Factors to treatments

Factors to Treatments

  • At least 1 Factor (IV) has to vary to have an experiment

    • Effect of screen size and input technique on performance (speed, accuracy)

  • An IV must always have at least 2 levels

  • Condition refers to a particular way that subjects are treated

    • Between subject: experimental conditions are the same as the groups

    • Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment)

Good experimental design

Good Experimental Design

  • Two-Group, Post-Test Design

  • Two conditions

  • Two groups:

    • Between subjects: random allocation

  • Treatment

  • Post-test: measure the DV

  • What’s really important?

Experimental designs

Experimental designs

  • Between subjects: Different participants - single group of participants is allocated randomly to the experimental conditions.

  • Within subjects: Same participants - all participants appear in both conditions.

  • Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.


Within subjects


  • Similar to the one-group pre-test-post-test design

  • It solves the individual differences issues

  • But raises other problems:

    • Need to look at the impact of experiencing the two conditions

    • Will they get tired? Gain practice? Learn what is expected?

  • Need to control for order and sequence effects?

Order effects

Order Effects

  • Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?)

  • Arises from warm-up, learning, fatigue, etc.

  • Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

Sequence effects

Sequence effects

  • Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2)

  • Effects viewed may not be main effects of the IV, but interaction effects

  • Can be controlled by arranging each condition to follow every other condition equally often



  • Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders

  • Self-directed learning: investigate the different counterbalancing methods

    • Randomization

    • Block Randomization

    • Reverse counter-balancing

    • Latin squares and Greco squares (when you can’t fully counterbalance)


Between within matched participant design

Between, within, matched participant design


Key points 1

Key points 1

  • Usability testing is done in controlled conditions.

  • Usability testing is an adapted form of experimentation.

  • Experiments aim to test hypotheses by manipulating certain variables while keeping others constant.

  • The experimenter controls the independent variable(s) but not the dependent variable(s).


  • Login