Today:

1. Today: • Assignment 2 back on Friday • True Experiments: Single-Factor Design • Today’s readings: • The base paper • What did you find in your domain of interest • Research paper

2. It’s a matter of control True Experiment Quasi Experiment Selection of subjects for the conditions Observe categories of subjects If the subject variable is the IV, it’s a quasi experiment Don’t know whether differences are caused by the IV or differences in the subjects • Random assignment of subjects to condition • Manipulate the IV • Control allows ruling out of alternative hypotheses

3. Other features • In some instances cannot completely control the what, when, where, and how • Need to collect data at a certain time or not at all • Practical limitations to data collection, experimental protocol

4. http://www.socialresearchmethods.net/kb/destypes.php

5. Validity • Internal validity is reduced due to the presence of controlled/confounded variables • But not necessarily invalid • It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences • Need to convince self and audience of the validity

6. External validity • If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting • Often comes down to what is most important for the research question • Control or ecological validity?

7. Terminology • Factors: Independent Variables (IVs) of an experiment • Level: particular value of an IV • Condition: a group or treatment (technique) • e.g., Condition 1: old system, Condition 2: new system • Treatment: a condition of an experiment • Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

8. Factors to Treatments • At least 1 Factor (IV) has to vary to have an experiment • Effect of screen size and input technique on performance (speed, accuracy) • An IV must always have at least 2 levels • Condition refers to a particular way that subjects are treated • Between subject: experimental conditions are the same as the groups • Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment) • Mixed: some variables are between, some within

9. Experimental designs • Between subjects: Different participants - single group of participants is allocated randomly to the experimental conditions. • Within subjects: Same participants - all participants appear in both conditions. • Takes care of individual differences • Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc. • Compromise – groups not likely to be equal, but can match on the factors you think might most impact results 9

10. Within-subjects • It solves the individual differences issues • But raises other problems: • Need to look at the impact of experiencing the two conditions • Will they get tired? Gain practice? Learn what is expected? • Need to control for order and sequence effects?

11. Order Effects • Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?) • Arises from warm-up, learning, fatigue, etc. • Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

12. Sequence effects • Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2) • Effects viewed may not be main effects of the IV, but interaction effects • Can be controlled by arranging each condition to follow every other condition equally often

13. Counterbalancing • Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders • Self-directed learning: investigate the different counterbalancing methods • Randomization • Block Randomization • Reverse counter-balancing • Latin squares and Greco squares (when you can’t fully counterbalance) • http://www.experiment-resources.com/counterbalanced-measures-design.html

14. Between, within, matched participant design 14

15. Images & additional notes text from: http://www.nationaltechcenter.org/index.php/products/at-research-matters/quasi-experimental-study/ True Experiment – Single Factor Design

16. Experimental Design: spot the flaw • One-Group Post-Test-Only Design • Group of subjects are given a treatment (x) • Single factor – only one IV • Then tested on the dependent variables (observation – o) • What’s the problem?

17. Experimental Design: spot the flaw • Post-Test-Only, non-equivalent control groups • Non-random (N) allocation of subjects into groups • One group is given the treatment, one doesn’t receive it (different levels to each group) • Post-test: measure the DV • What’s the problem?

18. Experimental Design: spot the flaw • One-Group Pre-Test-Post-Test Design • Single group (within subjects) • Pre-test: measure the DVs • Give the treatment • Post-test: Re-measure the DVs • What’s the problem?

19. Two-Group, Pre &Post-Test Design • Two groups: • Between subjects: random allocation • Treatment • Pre-test and Post-test: measure the DV

20. Within-subjects (repeated measures) • Similar to the one-group pre-test-post-test design • It solves the individual differences issues • But raises other problems: • Need to look at the impact of experiencing the two conditions • Will they get tired? Gain practice? Learn what is expected? • Need to control for order and sequence effects

21. Counter balanced

22. Determining effect of IV on the DV • Advantage of single factor design: • Easy to analyze (only one IV) • Fewer conditions (as many as number of levels) • Simple experimental design • BUT • Do not know how results would change for other levels of the controlled variables • Only measuring at one level of the controlled variable • Will the results hold for other levels? Or are there interactions between the IV and other variables?

23. Solutions? • Is repeating the experiment for another level of the controlled variable a valid solution? Why or why not?