1 / 40

Experimental Design

cree
Download Presentation

Experimental Design

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. 1 Experimental Design October 31, 2001 (none)(none)

    2. 2 John F. Kennedy “The greatest enemy of the truth is not the lie—deliberate, contrived, and dishonest—but the myth—persistent, pervasive, and unrealistic.” Quoted in Dawes, 1994, House of Cards Science Myths: The DARE Project keeps kids off drugs. Whole language teaches reading. “Keep warm or you’ll catch cold.”Science Myths: The DARE Project keeps kids off drugs. Whole language teaches reading. “Keep warm or you’ll catch cold.”

    3. 3 Advance Organizer Research Review Research Design: The Plan & Validity Internal Validity Statements of Causality External Validity Statements of Generalizability Designs Experimental Designs Quasi-experimental Designs Poor Designs Non-Experimental Designs Experimental Design I Research Review: 04—11 ~30 min. Research Design: 12—25 ~45 min. Internal Validity: 26—40 ~45 min. Experimental Design II External Validity: 03—20 ~45 min. Designs: 21—endExperimental Design I Research Review: 04—11 ~30 min. Research Design: 12—25 ~45 min. Internal Validity: 26—40 ~45 min. Experimental Design II External Validity: 03—20 ~45 min. Designs: 21—end

    4. 4 Research Why conduct research? To generate and acquire knowledge To answer research questions What is research? Refined decision-making process Method to answer important questions Does my intervention change behavior? Does a certain behavior (aggression) depend on a specific social skill (sharing)? Research: 04—11, ~30 minutesResearch: 04—11, ~30 minutes

    5. 5 A Few Decision Making Methods for Difficult Choices Status Quo—Irrelevant Criteria Flip a Coin or Let Time Decide Experience or Anecdote List Pros and Cons & Cross Off Recommended by Ben Franklin (see Dawes, 1988) “PROACT” Approach Hammond, Keeney, & Raiffa, 1999, Smart Choices Cost-Benefit Analysis Keeney & Raiffa, 1993, Decisions with Multiple Objectives Research—The Scientific Method Q: How many people have been taught how to make decisions? Examples: Which house do I buy? Where should we eat? Which parenting skills program should the Tri-Mountain Counseling Network adopt? Do our tobacco prevention videos reduce high school smoking initiation? These methods are neither mutually exclusive nor exhaustive. PROACT—a lecture in itself. Problems, Objectives, Alternatives, Consequences, & Tradeoffs. Also Uncertainty, Risk Tolerance, and Linked Decisions.Q: How many people have been taught how to make decisions? Examples: Which house do I buy? Where should we eat? Which parenting skills program should the Tri-Mountain Counseling Network adopt? Do our tobacco prevention videos reduce high school smoking initiation? These methods are neither mutually exclusive nor exhaustive. PROACT—a lecture in itself. Problems, Objectives, Alternatives, Consequences, & Tradeoffs. Also Uncertainty, Risk Tolerance, and Linked Decisions.

    6. 6 Experience Claims “In my experience, spanking is the only thing that will control my child.” “From my twelve years of teaching, I know that Hispanic children learn best when taught in Spanish.” “My grandfather smoked all his life, and he lived until he was 86!” “From my coaching experience . . .” “get the ball to the player with the hot hand.” “struggling shooters need to keep shooting.” Hot hand: see Dawes, 1988; Plous, 1993. Case Studies: many, but not all, case studies report the experiences of the researcher or the participant. Testimonials.Hot hand: see Dawes, 1988; Plous, 1993. Case Studies: many, but not all, case studies report the experiences of the researcher or the participant. Testimonials.

    7. 7 Experience Not “Experience keeps a dear school, but fools will learn in no other.” Benjamin Franklin, 1743 (See Dawes, 1988) Experience Gone Wrong Superstitions & baseless beliefs Cold Weather Causes Colds Graphology The Hot Hand Testimonials (better as examples) Case Studies (better for exploration) Other examples of baseless beliefs—beliefs without empirical evidence: Fear of flying. Some holistic medicines. Guns in the home make you safer. The Rorschach test. Suspensions improve behavior. Jinxing your favorite team by saying, “they’re gonna score.” Other examples of baseless beliefs—beliefs without empirical evidence: Fear of flying. Some holistic medicines. Guns in the home make you safer. The Rorschach test. Suspensions improve behavior. Jinxing your favorite team by saying, “they’re gonna score.”

    8. 8 Experience Not Conditions necessary to learn from experience Clear discrimination between correct and incorrect responses Immediate, unambiguous, & consistent consequences Problems with Experience Hindsight bias & selective memory No control; cannot account for luck or chance Cannot show Causality Can we learn from experience . . . How to teach reading? Whether my video series will reduce teen pregnancy? If parenting program X will reduce child problem behaviors? Other problems with experience: Confirmation bias. Correlation does not imply causation. Small (single-observation) sample. Self-fulfilling prophesies. Past experience changes future situations. No systematic feedback. No clear discrimination between correct and incorrect responses.Can we learn from experience . . . How to teach reading? Whether my video series will reduce teen pregnancy? If parenting program X will reduce child problem behaviors? Other problems with experience: Confirmation bias. Correlation does not imply causation. Small (single-observation) sample. Self-fulfilling prophesies. Past experience changes future situations. No systematic feedback. No clear discrimination between correct and incorrect responses.

    9. 9 Edgar L. Doctorow “When ideas go unexamined and unchallenged for a long time, certain things happen. They become mythological, and they become very, very powerful.” Quoted in Dawes, 1994, House of Cards (none)(none)

    10. 10 Research, A Better Option Review: Why conduct research? To generate and acquire knowledge To answer research questions: Does A cause B? Research Objectives Controlled: no alternate explanations Systematic: vary one thing at a time Replicable: repeatable by others Answers questions of causality Answers questions of causality only if conducted appropriately! Answers questions of causality only if conducted appropriately!

    11. 11 A Research Project: Seven Easy Steps Specify hypothesis, research question Review literature Design study—the Research Design Find someone to pay Collect data & carry out the intervention Analyze data Present results Steps in research: Specify hypothesis—discussed in class. Justify with literature summary—current homework project. Design research project—covered today and next week. Find money—Mike’s Grant Writing class. Collect data according to plan—after funding. Analyze—Stats courses. Publish results. Steps in research: Specify hypothesis—discussed in class. Justify with literature summary—current homework project. Design research project—covered today and next week. Find money—Mike’s Grant Writing class. Collect data according to plan—after funding. Analyze—Stats courses. Publish results.

    12. 12 “Research Design . . . “ “...the plan and structure of investigation, conceived so as to obtain answers to research questions” (Kerlinger & Lee, 2000, p. 449-450) Plan “Overall scheme or program of research” What you will do Structure “Model of relations among the variables” How you arrange what you want to study Research Design and Analysis are two different pieces: There is no such thing as an ANOVA design. Research Design: 12—25, ~45 minutes.Research Design and Analysis are two different pieces: There is no such thing as an ANOVA design. Research Design: 12—25, ~45 minutes.

    13. 13 Contents of the Plan Hypothesis Statement Design overview: timeline or design figure Participants: sampling & recruitment Intervention: theory & implementation Data collection: measures, procedures, & timing Intended statistical tests & power Critique: strengths & weaknesses Important Information! Statistical tests—the analyses—are just a small part of the whole research design.Important Information! Statistical tests—the analyses—are just a small part of the whole research design.

    14. 14 Example: the Cold & Colds Hypothesis: Cold temperatures will not increase the likelihood that a person will contract a a rhinovirus. Design: Two-Group Experimental Design Random assignment to two groups Expose all students to a cold virus Participants: 40 healthy UO students Intervention: place one group in cold room (Independent Variable: temperature of room) Data collection (dependent variable): mean number of cold symptoms, self report Hypothesis statement How would our experience answer research question? What are the hypothesis statements? Ho: Mc = Mw Ha: Mc ? Mw Design What if we assigned, say, members of two fraternities, in tact, to I & C? So why do we randomize? Participants: Why 40? People with colds before the study begins—need healthy people. Past exposure to the specific rhinovirus ? hence immunity. Data collection (Measurement). What is a cold symptom? That is, what is a runny nose? Allergy sufferers might confound the symptom counts.Hypothesis statement How would our experience answer research question? What are the hypothesis statements? Ho: Mc = Mw Ha: Mc ? Mw Design What if we assigned, say, members of two fraternities, in tact, to I & C? So why do we randomize? Participants: Why 40? People with colds before the study begins—need healthy people. Past exposure to the specific rhinovirus ? hence immunity. Data collection (Measurement). What is a cold symptom? That is, what is a runny nose? Allergy sufferers might confound the symptom counts.

    15. 15 Example: Colds (cont’d) Design Specifics & Figure Expose all Ss to a strain of rhinovirus Control: place students in a warm room Intervention: place in a “cold room” (50° F) Design Figure Intervention: R X O Control: R O X = exposure to cold room, 50° F, for two weeks O = number of cold symptoms R = random assignment DRAW A PICTURE!! Design figures: a picture is worth a thousand words. Why do we intentionally expose students to the cold? What if we thought our results might differ by age? Make sure each group has a range of ages: children, teens, . . ., retirees Split each group into subgroups, each with an age range: Children < 12 Teens College students 30 through 60 60 and aboveDRAW A PICTURE!! Design figures: a picture is worth a thousand words. Why do we intentionally expose students to the cold? What if we thought our results might differ by age? Make sure each group has a range of ages: children, teens, . . ., retirees Split each group into subgroups, each with an age range: Children < 12 Teens College students 30 through 60 60 and above

    16. 16 Example: Colds (cont’d) Analysis: statistical tests & power Compare groups on mean # of symptoms (M) Hypothesis: Ho: Mc = Mw vs. Ha: Mc <> Mw Analysis tool: independent t-test Power: if groups differ by an average of 2 symptoms, we have 80% chance of detecting it Critique: students who volunteer to catch a cold may differ from “normal” people Expected Results: Students in the “cold room” will contract the rhinovirus at the same rate of those in normal conditions Statistics Why an independent t-test? If we have no difference (real world), what are the chances that we’ll “find” one anyway? If we have a difference (real world), what is the chances that we don’t “find” one? What is an easy way to find no difference? Use a small sample. Statistics Why an independent t-test? If we have no difference (real world), what are the chances that we’ll “find” one anyway? If we have a difference (real world), what is the chances that we don’t “find” one? What is an easy way to find no difference? Use a small sample.

    17. 17 Research Design “The research design . . . can be thought of as a blueprint that provides the scientist with a detailed outline or plan for the collection and analysis of data.” Rosenthal & Rosnow, 1991, Essentials of Behavioral Research, p 69 Focus on the logic, not the details Controlled all these increase Systematic the validity of Replicable an experiment Oh, so many research designs. . . . Focus on the logic not the details.Oh, so many research designs. . . . Focus on the logic not the details.

    18. 18 What is Experimental Control? Not Controlled Cold room Ss sneak into warm room (a) A control student enters experiment with the flu & spreads it to all others (b) Increased Control Randomly assign Ss to balance chances of group differences before beginning (c) Use some kind of placebo so participants cannot tell which condition they’re in (d) Control leads to Validity General principle: We’re looking for factors that might influence how the two groups differ. Threats to Study Validity: Experimental treatment diffusion (i9) or contamination. History (i1). Controls for differential selection (i6). A true placebo may control for experimental treatment diffusion (i9), compensatory equalization of treatments (i11) or resentful demoralization of the control group (i12).General principle: We’re looking for factors that might influence how the two groups differ. Threats to Study Validity: Experimental treatment diffusion (i9) or contamination. History (i1). Controls for differential selection (i6). A true placebo may control for experimental treatment diffusion (i9), compensatory equalization of treatments (i11) or resentful demoralization of the control group (i12).

    19. 19 How to Limit Validity—Bad Language and learning, Span. vs. Eng. Ss taught in English get DI teacher; Ss taught in Spanish get whole language (e) The control participants, all in one room, complete final reading assessment sick (f) Intervention and control groups differ on socioeconomic status (g) Control parents learn about “treatment.” They buy a similar program from the Association for Direct Instruction (h) Move quickly! Threats to Study Validity: e) Multiple-treatment interference (e2)—a threat to external validity. f) Not a typical threat to internal validity, but probably falls under maturation (i2). g) Differential selection (i6). h) Compensatory rivalry by the control group (i10). Move quickly! Threats to Study Validity: e) Multiple-treatment interference (e2)—a threat to external validity. f) Not a typical threat to internal validity, but probably falls under maturation (i2). g) Differential selection (i6). h) Compensatory rivalry by the control group (i10).

    20. 20 How to Limit Validity—Bad Temperament training with one group Students selected because their parents said they “had problems” (i) Kids with more problems improve (j) Parents, to avoid embarrassment, learn how to look good on assessments (k) Families try extra hard after learning that they are in a research study (l) Move quickly! Threats to Study Validity: i) Generalization from the experimental sample to a defined population (p1). j) Statistical regression towards the mean (i5). k) Testing—they become “test wise” (i3). l) Hawthorne effect (e3).Move quickly! Threats to Study Validity: i) Generalization from the experimental sample to a defined population (p1). j) Statistical regression towards the mean (i5). k) Testing—they become “test wise” (i3). l) Hawthorne effect (e3).

    21. 21 Two Major Types of Validity Validity of Statements about Causality Can we draw conclusions about cause? Also called “internal validity” Validity of Statements about Generalization Can we expect the same results at other places or times, with other people, & with the intervention we reported? Also called “external validity” Internal Validity Coming Soon. History Maturation Testing Instrumentation Statistical Regression Differential Selection Experimental Mortality or Attrition Selection-Maturation Interaction Experimental Treatment Diffusion Compensatory Rivalry by the Control Group Compensatory Equalization of Treatments Resentful Demoralization of the Control Group Internal Validity Coming Soon. History Maturation Testing Instrumentation Statistical Regression Differential Selection Experimental Mortality or Attrition Selection-Maturation Interaction Experimental Treatment Diffusion Compensatory Rivalry by the Control Group Compensatory Equalization of Treatments Resentful Demoralization of the Control Group

    22. 22 How to Improve Validity—Good Maximize systematic variance Build a better intervention Minimize error variance Control extraneous variables Variables other than intervention that can affect an experiment’s outcome Examples of extraneous influence Wind, temperature in a physics experiment The time of day on a knowledge test (none)(none)

    23. 23 Maximize Systematic Variance Allow only systematic differences The only systematic difference between groups is the one planned in the study, usually the intervention Cold Example: treatment and control groups differ by only the temperature of the room & nothing else (none) (none)

    24. 24 Minimize Error Variance What is “error variance”? Unplanned or unsystematic differences between groups of participants Compare classroom vs. laboratory How do we reduce it? Improve measurement Increase sample size Increased sample size gives us an estimate closer to the “true” mean. Error may be completely unexpected; it might also not appear where expected. Anecdote from Rosenthal & Rosnow, 1991, Essentials of Behavioral Research, p 133: “Two forms of an intelligence test were administered to 171 [participants]. One form was administered in a quiet room. The second form was administered in a room with 7 bells, 5 buzzers, a 550-watt spotlight, 2 organ pipes of varying pitches, 3 metal whistles, a 55-point circular saw mounted on a wooden frame, a photographer taking pictures, and 4 students doing acrobatics! Remarkably, the grou did as well in the second situation ans in the first situation. Increased sample size gives us an estimate closer to the “true” mean. Error may be completely unexpected; it might also not appear where expected. Anecdote from Rosenthal & Rosnow, 1991, Essentials of Behavioral Research, p 133: “Two forms of an intelligence test were administered to 171 [participants]. One form was administered in a quiet room. The second form was administered in a room with 7 bells, 5 buzzers, a 550-watt spotlight, 2 organ pipes of varying pitches, 3 metal whistles, a 55-point circular saw mounted on a wooden frame, a photographer taking pictures, and 4 students doing acrobatics! Remarkably, the grou did as well in the second situation ans in the first situation.

    25. 25 Control Extraneous Variables Eliminate the extraneous variable Choose participants that are homogeneous with respect to the extraneous variable Choose setting, a time, place, or location, that does not change between participants Add extraneous variable to design as IV Split sample on a potentially extraneous variable Match participants on a potentially extraneous variable—some of each level into each group Ex: Education, completed high school, yes or no Randomize participants Eliminate: choose participants of the same age (cold example). Add: split participants on age (cold example).Eliminate: choose participants of the same age (cold example). Add: split participants on age (cold example).

    26. 26 Internal Validity “Internal validity refers to the degree of validity of statements made about whether X causes Y” Rosenthal & Rosnow, 1991, Essentials of Behavioral Research, p 64 The control of extraneous variables such that the observed effect can be attributed solely to the IV Gall, Borg, & Gall, 1996 See Cook & Campbell, 1979, Quasi-Experimentation, or Campbell & Stanley, 1963 Internal Validity: 26—40, ~45 min.Internal Validity: 26—40, ~45 min.

    27. 27 Threats to Internal Validity Extraneous Variables! History Maturation Testing Instrumentation Statistical Regression Differential Selection Experimental Mortality Selection-Maturation Interaction Experimental Treatment Diffusion Compensatory Rivalry by the Control Group Compensatory Equalization of Treatments Resentful Demoralization of the Control Group (none) (none)

    28. 28 History When study spans a period of time, it is possible that events occur, other than the IV, that influence the observed effects. In an intervention on steroid use with two intact teams, an intervention and a control, what history effects might arise over time? (none) (none)

    29. 29 Maturation When a study spans a period of time, it is possible that physical or psychological changes occur in participants that may influence observed effects. Students may experience gains in reading ability, math ability, or whatever simply because they have become developmentally older rather than because of exposure to the IV.Students may experience gains in reading ability, math ability, or whatever simply because they have become developmentally older rather than because of exposure to the IV.

    30. 30 Testing When a study employs a pre-post test (or multiple administrations of the same test), participants may learn about the test, or become “test wise” Testing problems might also occur from fatigue (none) (none)

    31. 31 Instrumentation Observed effects may result from a change in the characteristics of the measuring tool Judges may be initially more lenient Critics note that a recent reduction in drug use rates resulted from a decision to exclude homeless (none) (none)

    32. 32 Statistical Regression Test scores, over time, will move (regress) closer to the mean Extremely high scores will usually drop Extremely low scores will often rise A problem for single-group studies Youth corrections “treats” problem adolescents Kids enter system after exhibiting extreme behavior After they exit, they do better Did the system work, or is this regression towards the mean? (none) (none)

    33. 33 Differential Selection Participants differentially selected in to groups, either by accident or on purpose Control group members may choose to participate because they have no time for treatment activities (schools may constitute members) Differences may influence the observed influences of the IV (none) (none)

    34. 34 Experimental Mortality (Attrition) Occurs when participants drop out of a study before completion Threatens internal validity when participants drop out differentially across groups Study is focused on the performance of 1st grade classrooms using direct instruction reading vs. directed thinking learning activities (a whole language approach). At one school, first graders who are identified as slow readers are removed from their reading classes part way through the study to receive specialized instruction. At the other school, students with low ability remain in the class.Study is focused on the performance of 1st grade classrooms using direct instruction reading vs. directed thinking learning activities (a whole language approach). At one school, first graders who are identified as slow readers are removed from their reading classes part way through the study to receive specialized instruction. At the other school, students with low ability remain in the class.

    35. 35 Selection-Maturation Interaction Participants in each group differ with respect to maturation Differential selection when maturation is the confounding variable Depression study of adolescents. Early maturing girls more likely to get depressed. Late maturing boys more likely to get depressed.Depression study of adolescents. Early maturing girls more likely to get depressed. Late maturing boys more likely to get depressed.

    36. 36 Experimental Treatment Diffusion “I want what you have” phenomenon When treatment condition is considered favorable, participants in the control group may seek access to treatment Especially a problem when groups are close to each other or in the same setting (none) (none)

    37. 37 Compensatory Rivalry of the Control Group “I’m better than you are” phenomenon Control group performs more favorably than they would under normal circumstances because they perceive themselves as being in competition with the intervention group (none) (none)

    38. 38 Compensatory Equalization of Treatments If the goods and services in the intervention are perceived as desirable, interested others (e.g., administrators) may try to compensate the control group by providing them with equal goods and services. Parents of control students might try to provide extra help to their students, something comparable to what the intervention students get.Parents of control students might try to provide extra help to their students, something comparable to what the intervention students get.

    39. 39 Resentful Demoralization of Control Group If the control group thinks that a desirable intervention has been withheld from them, they may become discouraged and perform more poorly than normal. Class A placed on a group contingency to work toward a class pizza party once students have met math goals. Class B given only the normal “reinforcement” of their score. All students are asked to take a math test. Students in class B do not perform because they cannot earn a pizza party.Class A placed on a group contingency to work toward a class pizza party once students have met math goals. Class B given only the normal “reinforcement” of their score. All students are asked to take a math test. Students in class B do not perform because they cannot earn a pizza party.

    40. 40 Recap of Internal Validity History Maturation Testing Instrumentation Statistical Regression Differential Selection Experimental Mortality Selection-Maturation Interaction Experimental Treatment Diffusion Compensatory Rivalry by the Control Group Compensatory Equalization of Treatments Resentful Demoralization of the Control Group What is internal validity? The validity of statements about causality. Experimental control. Removal or accounting of extraneous variables. Ensuring observed effects can be attributed solely to the IV. What is internal validity? The validity of statements about causality. Experimental control. Removal or accounting of extraneous variables. Ensuring observed effects can be attributed solely to the IV.

    41. 41 Where Are We Now? Research Review Research Design: The Plan Internal Validity Statements of Causality External Validity Statements of Generalizability Designs Experimental Designs Quasi-experimental Designs Poor Designs Non-Experimental Designs (none) (none)

More Related