Chapter 3 Experiments, Quasi-Experiments, and Field Observations

1. Chapter 3Experiments, Quasi-Experiments, and Field Observations Winston Jackson and Norine Verberg Methods: Doing Social Research, 4e

2. The Rationale of the Experiment • John Stuart Mill — Method of Difference— the experiment is the key tool for discerning causal relations • A well-designed experiment should provide clear evidence of a cause-effect relation • Indicate whether or not the treatment variable (e.g., studying with the radio on) will bring about a change in some dependent measure (grade performance), other things being equal © 2007 Pearson Education Canada

3. Rationale (continued) • Internal validity: The extent to which the researcher can demonstrate that the treatment variable is having an impact on the dependent variable, and that other sources have been controlled • External validity: The extent to which the researcher can extrapolate the study findings to other groups in general © 2007 Pearson Education Canada

4. Key Elements in Experimental Designs • Dependent variable – the effect in a cause-effect relationship • Independent variable – the variable the researcher manipulates to determine whether and how it will change the dependent variable • the cause in a cause-effect relationship © 2007 Pearson Education Canada

5. Key Elements (cont’d) Kinds of independent variables: • Treatment Variables – variable studied • Control Variables – major influences intentionally controlled for in the experiment • Confounding Variables – variables that can unintentionally obscure or enhance results • Random Variables – vary without control, but are taken into account in study design (e.g., randomization) © 2007 Pearson Education Canada

6. Key Elements (cont’d) • Levels – often two or three levels • 2 x 2: two levels of the treatment variable and two levels in a control variable © 2007 Pearson Education Canada

7. Pseudo-Experimental Designs • Have limited scientific merit • Also called pre-experimental designs • Share some elements of classic experiment, however, they do not permit clear causal inferences • Two types: • Same group: pretest/post-test design • Exposed/comparison group design © 2007 Pearson Education Canada

8. A. Same Group: Pretest/Post-Test © 2007 Pearson Education Canada

9. Threats to Internal Validity • History – concurrent events • Maturation – changes in the individual subject • Testing – possible of response bias • Instrument Decay – unreliable measurement • Statistical Regression – extreme scores © 2007 Pearson Education Canada

10. Measures are taken at only one point in time. Problem: groups may not have been similar initially. The result may, or may not, be due to the treatment variable. B.Exposed/Comparison Group © 2007 Pearson Education Canada

11. More Threats to Internal Validity • Selection – Subjects selecting themselves into the study • Mortality – Subjects selecting themselves out of the study © 2007 Pearson Education Canada

12. Classic Experimental Designs • Two types: • Between-Subjects Design • Within-Subject Design • Both types of design allow a researcher to demonstrate causal inference © 2007 Pearson Education Canada

13. A. Between-Subjects Design © 2007 Pearson Education Canada

14. Between-Subjects Design (cont’d) • Involves a control and an experimental group • The experimental group is exposed to treatment intervention • The control group is exposed to neutral treatment © 2007 Pearson Education Canada

15. Key to Experimental Design • Construct treatment and control groups to be as similar as possible before the experiment begins. This is done by: • Randomization – each subject has an equal chance of being assigned to either group (provides control over both known [control] and unknown [random] factors) • Precision matching – matching subjects between groups • Combination of the above two methods © 2007 Pearson Education Canada

16. Key to Experimental Design (cont’d) • Blocking – Group subjects according to some controlled variable before randomly assigning them to a group • Baseline stability – Taking measures of the variable prior to introducing treatment © 2007 Pearson Education Canada

17. Analyzing the Data © 2007 Pearson Education Canada

18. Demonstrating a Causal Relation • Changes in treatment variable occur prior to changes in the dependent variable • The treatment and dependent variables are associated: as the treatment variable goes up, the dependent varies systematically • Nothing but the treatment variable has influenced the dependent variable © 2007 Pearson Education Canada

19. Ruling out Confounding Effects • Ensure that context is the same • Balance the background characteristics • Neutralize confounding (sources of spuriousness) variables • Deal with random variables © 2007 Pearson Education Canada

20. B. Within-Subject Designs • In the between-subjects design, the control for known and unknown factors is achieved through randomization • In the within-subject design, the control for known and unknown factors is achieved by exposing a subject to the different treatments • Since the subject is the same person, background characteristics, attitudes, and intelligence are all perfectly controlled • Also called control by constancy © 2007 Pearson Education Canada

21. Within-Subject Design (cont’d) • Subjects are exposed to the various treatments • Subjects’ own scores when exposed to different treatments are compared • Importance of having a baseline measure and returning to the original condition • The within-subject ABBA design: • A – measure dependent variable under original condition • B – measure dependent variable under treatment condition • B – continue treatment condition and measure dependent variable • A – measure dependent variable after returning to original condition © 2007 Pearson Education Canada

22. Hawthorne Effect • Refers to any variability in the dependent variable that is not the direct result of variations in the treatment variable • Hypothesis: worker productivity would increase as lighting intensity was increased • When lighting increased, productivity increased • HOWEVER, when lighting was later decreased, productivity did not decrease. WHY? • Interpretation: something other than treatment variable influenced workers – perhaps they worked faster because they knew were being observed © 2007 Pearson Education Canada

23. Quasi-Experimental Designs • Approximation of experimental design: done in situations where it is not possible to: • use random assignment • control the nature or timing of the treatment Example: • Henry & Ginzberg: Racial Discrimination in Employment (See Box 3.4, text pp. 75-77.) © 2007 Pearson Education Canada

24. Racial Discrimination in Employment • Two job applicants matched with respect to age, sex, education, physical appearance (dress), and personality were sent to apply for the same advertised job. • Only difference: one was White, one was Black • Results • Both offered job 5.0% • White offered job 13.4% • Black offered job 4.5% • Neither offered job 77.1% © 2007 Pearson Education Canada

25. Field Experiments • Researcher intervenes in a natural settings • Direct observations, “real” behaviour • Researcher intervention • Greeting stranger • Proxemics: norms surrounding personal space and the conditions under which such space will or will not be violated • Examples: cutting-through behaviour, greeting behaviour, helping behaviour © 2007 Pearson Education Canada

26. Naturalistic Observational Studies • Observe and record behaviour that occurs in a natural setting with those being observed unaware that they are being studied • Do not attempt to alter social environment • No intervention, simply record behaviour • Tally sheets are designed, then used to record the behaviour • Andrew Harrell’s Grocery Cart Safety study © 2007 Pearson Education Canada

27. Dressing for winter Parking violations Gender and smoking Professor/student participation: gender Seat belt compliance Speeding Antigonish Buying healthy food ABM behaviour Termination of conversations Drinking patterns Smoking behaviour in teens Stop sign Tipping Samples of Student Research Projects © 2007 Pearson Education Canada

28. Steps in Doing Study • Restrict observations • Review of literature • Develop hypotheses • Define terms • Develop a tally sheet (See Figure 3.5, p. 90) • Transfer data to master table (see Figure 3.6, p. 90) • Creating tables (Tables 3.4, 3.5, 3.6, p. 91) • Writing the report © 2007 Pearson Education Canada

29. Field and Observational Studies: An Assessment • Weak on generalizations • Strong on validity (real behaviour) • Making causal inferences a challenge • Multivariate a problem • Probing strong with participant observation, in-depth interviews, and focus groups • Probing weak with naturalistic observational © 2007 Pearson Education Canada

30. Advantages and Disadvantages of Experimental Designs • Advantages: • Ease of making clear causal inferences • Disadvantages: • Low external validity: poor on generalization to a larger population • Concerns about the artificiality of lab • Poor on probing, poor on multivariate • Experiments cannot study all topics © 2007 Pearson Education Canada