Experimental and Causal-Comparative Designs

1. Experimental and Causal-Comparative Designs

2. Purpose • Examine the possible influences that one factor or condition may have on another factor or condition • cause-and-effect relationships • ideally, by controlling all factors except those whose possible effects are the focus of investigation

3. What is Experimentation? • Why do events occur under some conditions and not under others? • Research methods that answers these questions are called causal methods • ex post facto research designs - observes what is or what has been, also has the potential for discovering causality, but researcher is required to accept the world as found • experiment allows the researcher to alter systematically the variables of interest and observe what changes follow

4. Experiments • Studies involving intervention by the researcher beyond that required for measurement • The researcher manipulates the independent or explanatory variable and then observes whether the hypothesized dependent variable is affected by the intervention

5. Example of Bystanders and Thieves • Students were asked to an office where they had an opportunity to see a fellow student steal some money from a receptionist’s desk. A confederate of the experimenter, did the stealing. The hypothesis concerned whether people observing a theft would be more like to report it (1) if they saw the crime alone or (2) if they were in the company of someone else.

6. Variables in the Study • Independent - was the state of either being alone when observing the theft or being in the company of another person. • Dependent - whether the subjects reported observing the crime • the results indicated that people were more likely to report the theft if they observed it alone rather than in another person’s company

7. How did the researchers come to this conclusion? • first there must be an agreement between the independent and dependent variables • the presence or absence of one is associated with the presence or absence of the other • more reports of the theft came from lone observers than from paired observers

8. How did they come to this conclusion? • second, the time order of the occurrence of the variables must be considered. • The dependent variable should proceed the independent variable. • It is unlikely that people could report a theft before observing it

9. How did they come to this conclusion? • Third - researchers are confident that other extraneous variables did not influence the dependent variable • researchers controlled their ability to confound the planned comparison • the event was staged without the observer’s knowledge • only the receptionist, observers, and the “criminal” were in the office • the same process was repeated with each trial

10. Conducting an Experiment • Experiment is the premier scientific methodology for establishing causation • however the resourcefulness and creativeness of the researcher are needed to make the experiment live up to its potential • to make it successful the researcher must plan carefully

11. Seven Activities to Accomplish • Select relevant variables • Specify the level(s) of treatment • Control the experimental environment • Choose the experimental design • Select and assign the subject • Pilot-test, revise and test • Analyze the data

12. Selecting Relevant Variables • It is the researcher’s task to translate an amorphous problem into the hypothesis that best states the objectives of the research • hypothesis is a relational statement because it describes a relationship between two or more variables • researcher must select variables that are the best operational representation of the original concepts

13. Specifying the Levels of Treatment • Treatment levels of the independent variable are the various aspects of the treatment condition. • For example, if education was hypothesized to have an effect on employment stability, it might be divided a high-school, college, graduate • based on simplicity and common sense • alternatively a control group could provide a base level for comparison

14. Controlling the Experimental Environment • The potential for distorting the effect of treatment on the dependent variable must be controlled • examples : videotaping instructions, arrangement of room, time of administration, experimenter’s contact with subjects

15. Choosing the Experimental Design • Experimental design serves as positional and statistical plans to designate relationships between experimental treatment and the experimenter’s observations or measurement points

16. Selecting and Assigning Subjects • Represent the population to be generalized • random assignment • matching - each experimental and control subject match

17. Pilot Testing, Revising and Testing • Pilot test - reveal errors in design • refinements

18. Analyzing the Data • If planning and pretesting have occurred, experimental data will take an order and structure.

19. Validity in Experimentation • Always a question if the results are true • internal validity - do the conclusions we draw about the demonstrated experimental relationship truly imply cause? • External validity - does an observed causal relationship generalize across person, settings and times

20. Internal Validity • History • during the time an experiment is taking place, some events may occur that confuse the relationship being studied • take a control measurement (O1) of the dependent variable before introducing the manipulation (X), after the manipulation we take an after measurement (O2) of the dependent variable. Then the difference between O1 and O2 is the change that the manipulation caused

21. Maturation • Changes occur within the subject that of the function of the passage of time and not specific to any particular event • special concern when study covers a long time • hunger, bored, tired are also factors in shorter test

22. Testing • The process of taking a test can affect the scores of a second test • the more experience of taking the first test can have a learning effect that influences the results of the second test

23. Instrumentation • Changes between observations • using different questions at each measurement • using different observers or interviewers • observer experience, boredom, fatigue and anticipation of results can all distort the results of separate observations

24. Selection • Differential selection of subjects for experimental and control group. • Groups must be equivalent in every respect • if subjects are randomly assigned to experimental and control groups, the selection problem can be largely overcome

25. Statistical Regression • This factor operates especially when groups have been selected by their extreme scores • suppose we only take the workers with top 25% and bottom 25% of productivity scores • no matter what is done between O1 and O2 there is a strong tendency for the average of the high scores at O1 to decline at O2 and for the low scores at O1 to increase • In the second measurement, members of both groups score more closely to their long-run mean scores

26. Experiment Mortality • Composition of the group changes during the test • attrition - people dropout • because members of the control group are not affected by the testing situation, they are less likely to withdraw • diffusion or imitation of treatment - if the people in control and experimental group talk, they learn of the treatment eliminating the difference between the group

27. Experiment Mortality • Compensatory equalization - the experimental treatment is much more desirable, there may be an administrative reluctance to deprive the control group members • Compensatory rivalry - when members of the control group know they are the control group. This may generate competitive pressures causing them to try harder

28. Experiment Mortality • Resentful demoralization of the disadvantage - when the treatment is desirable and the experiment is obtrusive, control members may become resentful of their deprivation and lower their cooperation and output • local history - when one assigns all experimenters to one group and all control people to another - there can be idiosyncratic events that may confound

29. External Validity • Internal validity factors cause confusion about whether the experimental treatment (X) or extraneous factors are the source of observation differences. • In contrast, external validity is concerned with the interaction of the experimental treatment with other factors and the resulting impact on abilities to generalize to times, settings, or persons

30. The Reactivity of Testing on X • Is one of sensitizing subjects by the pretest so they respond to the experimental stimulus in a different way. • A before measurement of the level of knowledge about the ecology programs of a company will often sensitize the subject to the various experimental communication efforts that might then be made about the company

31. Interaction of Selection of X • The process by which test subject are selected • the population from which one selects subjects may not be same as the population to which one wishes to generalize the results

32. Other Reactive Factors • Experimental setting themselves may have a biasing effect on the subject’s response to X • if subjects know they are participating, they may have a tendency to role-play • external validity may be hard to control because it is a matter of generalization • try and secure as much internal validity requirements

33. Experimental Research Designs • Many • vary widely in their power to control contamination of the relationship between independent and dependent variables • the most widely accepted designs are based on this characteristic of control: • preexperiments • true experiments • field experiments

34. Key to Design Symbols • X - an X represents the introduction of an experimental stimulus to a group. The effects of this independent variable(s) are of major interest • O - an O identifies a measurement or observation activity • R - an R indicates that the group members have been randomly assigned to a group.

35. Keys to Timing • The X’s and O’s in the diagram are read from left to right in temporal order. • O X O O • X’s and O’s vertical to each other indicate that the stimulus and or observation take place simultaneously O X X

36. Keys to Selection • Parallel rows that are not separated by dashed lines indicate that comparison groups have been equalized by the random process • those separated with a dashed line have not been so equalized X O O X O O O

37. Seven Activities to Accomplish • Select relevant variables • Specify the level(s) of treatment • Control the experimental environment • Choose the experimental design • Select and assign the subject • Pilot-test, revise and test • Analyze the data

38. Experimental Designs

39. Preexperimental Designs • One-Shot Case Study • One-Group Pretest-Posttest Design • Static Group Comparison • All three are weak in their scientific measurement power because they fail to control the various threats to internal validity. This is especially true of the one-shot case study.

40. X Treatment or manipulation of independent variable O Observation or measurement of dependent variable One-Shot Case Study An example is an employee education campaign about new technologies without prior measurement of employee knowledge. Results would reveal only how much the employees know after the campaign, but there is no way to judge the effectiveness of the campaign. The lack of pretest and control group make this design inadequate for establishing causality.

41. One-Group Pretest-Posttest Design O X O Pretest Manipulation Posttest Can be used for the educational example, but how well does it control for history? Maturation? Testing effect?

42. Static Group Comparison X O1 O2 This design provides for two groups, one of which receives the experimental stimulus while the other serves as a control. A forest fire or other natural disaster is the experimental treatment, and the psychological trauma (or property loss) suffered by the residents is the measured outcome. A pretest before the fire would be possible … but. The control group, receiving the posttest, would consist of residents whose property was spared. Weakest link, no way certain that the two groups are equivalent.

43. True Experimental Designs • Major deficiency of the preexperimental designs is they fail to provide comparison groups that are equivalent. • The way to achieve equivalence is through matching and randomization. • Two Classical • Pretest-Posttest Control Group Design • Posttest-Only Control Group Design

44. Pretest-Posttest Control Group Design R O1 X O2 R O3 O4 The effect of the experimental variable is E = ( O2 –O1 ) – ( O4 –O3 ) In this design, the seven major internal validity problems are dealt with fairly well, although there are still some difficulties. Local history may occur in one group and not the other, communication between people in test and control groups, and mortality.

45. Solomon Four-Group Design R O1 X O2 R O3 O4 R X O5 R O6 The addition of the two groups that are not pretested provides a distinct advantage. If the researcher finds that O5 and O do not differ from the top two groups observation, the researcher can generalize findings to situations where no pretest was given. The Solomon Four-Group Design enhances the external validity

46. Posttest-Only Control Group Design R X O1 R O2 In this design the pretest measurements are omitted. Pretests are not really necessary when it is possible to randomize. Experimental effect is ( O1 – O2 ) Since the subjects are measured only once, the threats of testing and instrumentation are reduced.

47. Extensions of True Experimental Designs • Those which were discussed were classical design forms, but researchers normally use an operational extension of the basic design in • The number of different experimental stimuli that are considered simultaneously by the experimenter • The extent to which assignment procedures are used to increase precision

48. Factor • Widely used to denote an independent variable • May be divided into treatment levels, which represent subgroups • Active factors – are those that the experimenter can manipulate by causing a subject to receive one level or another • Blocking factor – can only identify and classify the subject on an existing level (gender,age,organizational rank)

49. Completely Randomized Design R O1 X1 O2 R O3 X2 O4 R O5 X3 O6 Experiment: to determine the ideal difference in price between a store’s private brand of vegetables and national brands. There will be three price spreads (treatment levels) of 7, 12 and 17 cents. 18 stores are randomly divided (6 to each treatment group). The price differential is maintained for a period and then a tally is made of the sales volumes and gross profit of the cans for each group of stores.

50. Randomized Block Design The critical reason for randomize block design is that the sample size is too small that is risky to depend on random assignment alone. Small samples such as 18 stores are typical in field experiments because of high costs. Another reason for blocking is to learn whether treatments bring different results among various groups of subjects. Assume there is reason to believe that lower-income families are more sensitive to price differentials than are higher-income families. This factor could seriously distort our results unless we stratify the stores by customer income.