RESEARCH DESIGN

1. RESEARCH DESIGN PSY 4603 Research Methods

2. RESEARCH DESIGN Experimental Designs The specific research designs available to investigators can be divided into two basic types: • group designs, and • single-subject designs.

3. RESEARCH DESIGN A Typical Experimental Design Pretest-Posttest Control Group Design R O1 X O2 R O3 O4

4. Experiments are studies involving intervention by the researcher beyond that required for measurement. • The usual intervention is to manipulate some variable in a setting and observe how it affects the subjects being studied (e.g., people or physical entities). • The researcher manipulates the independent or explanatory variable and then observes whether the hypothesized dependent variable is affected by the intervention.

5. Experiments • There is at least one independent variable (IV) and one dependent variable (DV) in a causal relationship. • An example of such an intervention is the study of bystanders and thieves. • In this experiment, students were asked to come to an office where they had an opportunity to see a fellow student steal some money from a receptionist's desk. • A confederate of the experimenter, of course, did the stealing. • The major hypothesis concerned whether people observing a theft would be more likely to report it (1) if they observed the crime alone or (2) if they were in the company of someone else.

6. Experiments • We hypothesize that in some way the IV "causes" the DV to occur. • The independent or explanatory variable in our example was the state of either being alone when observing the theft or being in the company of another person. • The dependent variable was whether the subjects reported observing the crime. • The results suggested that bystanders were more likely to report the theft if they observed it alone rather than in another person's company.

7. On what grounds did the researchers conclude that people who were alone were more likely to report crimes observed than people in the company of others? • Three types of evidence form the basis for this conclusion. • First, there must be an agreement between independent and dependent variables. • The presence or absence of one is associated with the presence or absence of the other. • Thus, more reports of the theft (DV) came from lone observers (IV1) than from paired observers (IV2). • Second, beyond the correlation of independent and dependent variables, the time order of the occurrence of the variables must be considered. • The dependent variable should not precede the independent variable. They may occur almost simultaneously, or the independent variable should occur before the dependent variable. • This requirement is of little concern since it is unlikely that people could report a theft before observing it. • The third important support for the conclusion comes when researchers are confident that other extraneous variables did not influence the dependent variable. • To ensure that these other variables are not the source of influence, researchers control their ability to confound the planned comparison. • Under laboratory conditions, standardized conditions for control can be arranged. • The crime observation experiment was carried out in a laboratory set up as an office. • The entire event was staged without the observers' knowledge. The receptionist whose money was to be stolen was instructed to speak and act in a specific way. • Only the receptionist, the observers, and the "criminal" were in the office. • The same process was repeated with each trial of the experiment.

8. An Evaluation of Experiments • Advantages of an Experiment • When I elaborated on the concept of cause, I said causality could not be proved with certainty but the probability of one variable being linked to another could be established convincingly. • The experiment comes closer than any primary data collection method to accomplishing this goal. • The foremost advantage is the researcher's ability to manipulate the independent variable.

9. Also, a control group serves as a comparison to assess the existence and potency of the manipulation. • The second advantage of the experiment is that contamination from extraneous variables can be controlled more effectively than in other designs. • Third, the convenience and cost of experimentation are superior to other methods. • Fourth, replication-repeating an experiment with different subject groups and conditions-leads to the discovery of an average effect of the independent variable across people, situations, and times. • Fifth, researchers can use naturally occurring events and, to some extent, field experiments to reduce subjects' perceptions of the researcher as a source of intervention or deviation in their everyday lives.

10. Disadvantages of an Experiment • The artificiality of the laboratory is arguably the primary disadvantage of the experimental method. • Second, generalization from nonprobability samples can pose problems despite random assignment. • Third, despite the low costs of experimentation, many applications of experimentation far outrun the budgets for other primary data collection methods. • Fourth, experimentation is most effectively targeted at problems of the present or immediate future. • Finally, management research is often concerned with the study of people. • There are limits to the types of manipulation and controls that are ethical.

11. Experimental, Quasi-Experimental, and Ex Post Facto (Causal-Comparative) Research

12. Characteristics of Experimental Research • There is a control or comparison group • Subjects are randomly assigned to groups • The treatment is randomly assigned to groups.

13. Characteristics of Quasi-Experimental Research • There is a control or comparison group • Intact groups are used • The treatment is randomly assigned to groups.

14. Characteristics of Ex Post Facto Research • There is a control or comparison group • Intact groups are used • The treatment is not manipulated, it has already occurred.

16. Diagramming Research • To illustrate research designs, a number of symbols are used • X1 = Treatment • X2 = Control Group • O = Observation (pretest or posttest) • R = Random Assignment

17. A Sample Research Design • Single-Group Pretest-Treatment-Posttest Design R O X1 O This means subjects are randomly assigned to a group, which is then given a pretest, then there is a treatment, then there is a posttest.

18. R O X1 O • This is not really an experimental design because there is no control group • It is often referred to as a preexperimental design • Novice researchers often use this research design • There are some major problems with this design – did the treatment really make the difference or was something else happening.

19. R O X1 O • What are the threats to the Internal Validity of this type of research (Did the treatment really cause a difference?)

20. Internal Validity Threats • History • Another event occurs during the time of the experiment that might cause the difference • An experiment to heighten racial awareness was conducted by a researcher during February. This is Black History month; so the results might be affected by events that occur during Black History month and not the treatment.

21. Internal Validity Threats • Maturation • People naturally change and evolve over time. This may cause the difference. • A college develops a new housing plan to promote more open-mindness and acceptance of others. The students are tested when they enter college and when they graduate. The results show they are now more open-minded and tolerant of others. Did the housing plan work or do students just mature and grow as a result of the college experience.

22. Internal Validity Threats • Mortality • Some people drop out during an experiment. This may affect the outcome. • I am teaching a new experimental seminar on study skills. About half of the class stopped coming to the seminar before the semester was over. The students who remained improved their study skills. So my course was effective! • Probably not. The half that stopped coming might not have gained anything; that is why they stopped attending.

23. Internal Validity Threats • Testing • Whenever you give a pretest, the students may remember the test questions, and get them correct on the posttest. • I gave a test to my study skills group on Monday, presented some unique concepts on Tuesday, then gave them the posttest on Wednesday. The grades were significantly higher on the posttest. • It is possible the grades were higher because the students still remembered the questions from the pretest.

24. Internal Validity Threats • Instrumentation • To overcome the testing threat to internal validity, a researcher develops a different form of the test instrument, but it is not really equivalent. • I gave a test to my study skills group on Monday, presented some unique concepts on Tuesday, then gave them an alternative form of the pretest on Wednesday. The grades were significantly higher on the posttest. • It is possible the grades were higher because the second test was easier than the first.

25. Internal Validity Threats • Regression • When subjects are selected because of extreme scores on some type of instrument, there is tendency for their scores to move more toward the average on subsequent tests. • An experimenter selected students for a reading program based on their low test scores. At the end of the treatment, the test scores had improved. • Extreme scores naturally move toward the mean on subsequent tests.

26. How to Handle Internal Validity Threats • Have a control group and use randomization.This design is the Two-Group Pretest-Treatment-Posttest Design. The Control Group would experience the same history and maturation. Mortality should be the same because of random assignment. Random assignment eliminates the selection threat. However testing and instrumentation could still be a threat. R O X1 O R O X2 O

27. Other Research Designs • Two-Group Treatment-Posttest-Only Design There is no pretest so this eliminates the testing and instrumentation threat to internal validly but you don’t know about their knowledge or attitude coming into the study. R X1 O R X2 O

28. Other Research Designs • Solomon 4-Group Design Note: A blank indicates the control group, same as X2 R O X1 O R X1 O R O O R O

29. Quasi-Experimental Designs • Posttest Only Nonequivalent Group Design The absence of R indicates there is no random assignment. Sometimes you will see a dotted line between the two groups. This indicates the two groups may not be equivalent. X1 O X2 O

30. Quasi-Experimental Designs • Pretest-Posttest Nonequivalent Group Design O X1 O O X2 O

31. External Validity • Can the research be generalized to other settings? • Population Validity • Personological Variables • Ecological Validity

32. Population Validity • Is the sample population similar to the population the researchers wishes to generalize to

33. Individual Differences (Personological Variables) • Different people have different personalities, learning styles, etc., so the results may not be generalizable to people who are substantially different on these personological variables.

34. Ecological Validity • The setting or situation in which the experiment occurred may be different than other settings.

35. Social Interaction Validity Threats • Diffusion or Imitation of Treatment • This occurs when a comparison group learns about the program either directly or indirectly from program group participants. • This group may try to imitate or emulate what the treatment group is getting.

36. Social Interaction Validity Threats • Compensatory Rivalry • The comparison group knows what the program group is getting and develops a competitive attitude with them.

37. Social Interaction Validity Threats • Resentful Demoralization • This is almost the opposite of compensatory rivalry. Here, students in the comparison group know what the program group is getting. But here, instead of developing a rivalry, they get discouraged or angry and they give up.

38. Social Interaction Validity Threats • Compensatory Equalization of Treatment • The researcher is under pressure to “enrich” the experiences of the control group. This pressure may come from parents, school administrators, etc.

39. Ex Post Facto (Quasi-Experimental) Research • Explores possible causes and effects • The independent variable is not manipulated, it has already been applied • Focuses first on the effect, then attempts to determine what caused the observed effect.

40. RESEARCH DESIGN Group Designs The group (multi-subject) designs all include one or more groups of subjects and are classified as either: • between-groups, • within-subjects, • or mixed.

41. Between-Groups Design Between-groups design is used to assess the effects of different levels of an independent variable by administering each level to a different group of subjects and then comparing the status or performance of the group on the dependent variable. • The simplest between-groups designs include a single independent variable with two levels. When using the design, the study includes two groups that each receives a different level of the IV.

42. RESEARCH DESIGN Example: A psychologist assesses the effects of a “self-control” procedure by comparing the achievement of children who have been trained in the procedure (experimental group) with that of children who have not been trained in the procedure (control group).

43. RESEARCH DESIGN The simple two-group design can be expanded in two ways. One way is to include more than two levels of a single independent variable. The psychologist in this study could compare three levels of the control procedure • a procedure that includes self-instruction only, • a procedure that includes self-instruction, self-monitoring, and self-reinforcement, and • no procedure. In this situation, the study would involve comparing the average academic achievement test scores of subjects in the three groups.

44. RESEARCH DESIGN Another way to expand a two-group design is to include two or more independent variables. Whenever a study includes two or more independent variables, it is called a factorial design. The major advantage of a factorial design is that it provides more thorough information about the relationships among variables by allowing an investigator to analyze the main effects of each independent variable as well as the interaction between independent variables.

45. RESEARCH DESIGN If the psychologist in the self-control study included initial symptom severity (mild, moderate, and severe) as a second independent variable, s/he would be able to determine if there are: • MAIN effects of the self-control procedure, • MAIN effects of initial symptom severity, and/or • an INTERACTION between self-control procedure and initial symptom severity.

46. RESEARCH DESIGN Main Effect A main effect is the effect of one independent variable on the dependent variable, disregarding the effects of all other independent variables. An interaction refers to the effects of two or more independent variables considered together. • An interaction occurs when the effects of an independent variable differ at different levels of another independent variable.

47. RESEARCH DESIGN Illustration: Assume the psychologist in the self-control study obtains a sample of 60 children and divides them into three groups on the basis of their initial symptom severity (mild, moderate, or severe). S/he then randomly assigns subjects in each group to either the experimental (self-control procedure) or control (no procedure) group so that there are 10 children in each of the study's now six groups (see table in next slide) • Although the data collected by the psychologist would have to be analyzed with an inferential statistical test to determine if there are significant main and/or interaction effects, tentative conclusions can be drawn by examining the data.

48. Self-Control Procedure No Procedure Mild Symptoms 52 36 Moderate Symptoms 40 30 Severe Symptoms 34 36 RESEARCH DESIGN As an example, assume that the psychologist obtains the following mean achievement test scores for the six groups of children:

49. RESEARCH DESIGN To determine if there are main effects of each IV, it is necessary to calculate the marginalmeans. For self-control procedure, the marginal means are 42 and 34. These means were obtained by adding the means in each column and dividing by 3 (the number of means): (52 + 40 + 34)/3 = 42 and (36 + 30 + 36)/3 = 34. • Because the marginal means are different, it is possible to tentatively conclude that there are main effects for the self-control procedure. • Overall, the self-control procedure seems to have beneficial effects on academic achievement test scores.

50. RESEARCH DESIGN For initial symptom severity the marginal means are 44, 35, and 35. These means were obtained by adding the means in each row and dividing by 2 (the number of means): (52 + 36)/2 = 44; (40 + 30)/2 = 35; and (34 + 36)/2 = 35. • The marginal means indicate that there are also main effects for initial symptom severity. • Although children with moderate and severe symptoms obtained the same mean achievement test score (35), children with mild symptoms obtained a higher mean score (44). (If all three means were identical, there would be no main effects of symptom severity.) • This indicates that, overall, mild symptoms are associated with the highest achievement test scores.