Matters of Design

Matters of Design PowerPoint PPT Presentation


  • 194 Views
  • Uploaded on
  • Presentation posted in: General

Internal Validity. Internal validity ? the extent to which we can accurately state that the IV produced the observed effect. External Validity. External validity ? the extent to which the results of an experiment can be applied to and across different persons, settings, and times. Is it more important to be able to generalize the results to other populations, settings, etc.?.

Download Presentation

Matters of Design

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


1. Matters of Design

2. Internal Validity Internal validity – the extent to which we can accurately state that the IV produced the observed effect The primary goal of the researcher in designing an experimental study is to attain internal validity. A second, and almost equally important goal is to attain external validity. Define internal validity – the ability to draw conclusions about causal relationships from the data You cannot establish cause and effect without internal validity. Study is worthless without it. However, the problem is determining that the “effect” was caused only by the IV. Therefore, you must control confounding variables that vary systematically with the IV. If you know that a variable does not vary systematically with IV, then you don’t have to worry about it. This is an example of using logic vs. control in terms of designing your study. The problem is that strong internal validity is often at odds with strong external validity.The primary goal of the researcher in designing an experimental study is to attain internal validity. A second, and almost equally important goal is to attain external validity. Define internal validity – the ability to draw conclusions about causal relationships from the data You cannot establish cause and effect without internal validity. Study is worthless without it. However, the problem is determining that the “effect” was caused only by the IV. Therefore, you must control confounding variables that vary systematically with the IV. If you know that a variable does not vary systematically with IV, then you don’t have to worry about it. This is an example of using logic vs. control in terms of designing your study. The problem is that strong internal validity is often at odds with strong external validity.

3. External Validity External validity – the extent to which the results of an experiment can be applied to and across different persons, settings, and times The problem is that strong internal validity is often at odds with strong external validity. Define external validity. Control means a laboratory, out of the natural setting – therefore, generalizability is compromised. What is the solution? Acknowledge that no one experiment will meet all conditions. Identify which is more important to the question (EV or IV) and design accordingly. When possible, plan a series of experiments that start with strong internal validity and move towards strong external validity. Identify limitations of design and interpret results accordingly. The problem is that strong internal validity is often at odds with strong external validity. Define external validity. Control means a laboratory, out of the natural setting – therefore, generalizability is compromised. What is the solution? Acknowledge that no one experiment will meet all conditions. Identify which is more important to the question (EV or IV) and design accordingly. When possible, plan a series of experiments that start with strong internal validity and move towards strong external validity. Identify limitations of design and interpret results accordingly.

4. What do we mean by control? Exerting a constant influence. Eliminate the variable completely if possible Eliminate any differential influence that variable may have across the levels of IV Keep the influence of EV constant across levels of IV Easy for some variables (gender, how instructions are given) Difficult to do with certain variables (motivation, fatigue, learning ability, understanding of certain words) because we can’t measure them precisely enough or because they change as the experiment progresses Eliminate the variable completely if possible Eliminate any differential influence that variable may have across the levels of IV Keep the influence of EV constant across levels of IV Easy for some variables (gender, how instructions are given) Difficult to do with certain variables (motivation, fatigue, learning ability, understanding of certain words) because we can’t measure them precisely enough or because they change as the experiment progresses

5. Control Techniques Randomization Random selection Random assignment Experimental control Elimination Make IV Matched groups Control group

6. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

7. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

8. Experimenter Expectancy Experimenter bias (expectancy effect, Pygmalion effect, self-fulfilling prophecy effect) Solution: Training & practice Run all conditions simultaneously Automated procedures (videotape, computer scoring) Blind the investigator Error introduced as a result of the expectations of the experimenter May occur in many ways: Subjects may perform better because of the researcher’s (subtly or clearly) expressed expectations about a certain intervention Investigator may (subtly or clearly) influence or “coach” subject’s (from a certain group) performance during data collection Investigator may make scoring errors when recording data Pygmalion effect – effects caused by subjects’ perceptions of the expectations of the researcher; subjects on a certain intervention improve because the experimenter believes that that intervention is best Solution – Blind the investigator. Blind everyone who has interaction with the subjects –raters, observers, scorers, data collectors, treatment administrators, etc. The investigator does not know which treatment group a subject belongs to.Error introduced as a result of the expectations of the experimenter May occur in many ways: Subjects may perform better because of the researcher’s (subtly or clearly) expressed expectations about a certain intervention Investigator may (subtly or clearly) influence or “coach” subject’s (from a certain group) performance during data collection Investigator may make scoring errors when recording data Pygmalion effect – effects caused by subjects’ perceptions of the expectations of the researcher; subjects on a certain intervention improve because the experimenter believes that that intervention is best Solution – Blind the investigator. Blind everyone who has interaction with the subjects –raters, observers, scorers, data collectors, treatment administrators, etc. The investigator does not know which treatment group a subject belongs to.

9. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

10. Subject Expectancy Hawthorne effect (placebo effect, guinea pig effect, novelty effect, gee whiz effect) Rating Effect (Halo effect, overrater error, underrater error, central tendency error) Avis Effect John Henry effect Error introduced as a result of the expectations of the subjects Hawthorne effect (guinea pig effect, novelty effect, gee whiz effect) effects caused by subjects’ feeling like they are special just because they are in the study and possibly getting attention; affects all treatment groups (placebo & experimental, not just group actually receiving treatment) Any effect of a treatment of any kind caused by the subjects’ belief that an effect should be expected (e.g., O2 on sidelines) Placebo effect – subject believes that “placebo” (pill, injection, or other “treatment” that contains not active ingredient) is helping them just because they are being treated Rating Effect (Halo effect, overrater error, underrater error, central tendency error) tendency toward evaluative bias resulting from pre-existing attitudes or opinions toward whomever or whatever is being evaluated; subjects may positively or negatively evaluate something because of a previous experience with the company or product or lab solution is to select subjects without prior association (juries) or to collect additional data that might indicate direction or strength of halo effect Avis Effect (opposite of John Henry effect) effects caused by control group perceiving that they are second fiddle; subjects in control group tend to try harder Inform them of the significance of their role John Henry effect Error introduced as a result of the expectations of the subjects Hawthorne effect (guinea pig effect, novelty effect, gee whiz effect) effects caused by subjects’ feeling like they are special just because they are in the study and possibly getting attention; affects all treatment groups (placebo & experimental, not just group actually receiving treatment) Any effect of a treatment of any kind caused by the subjects’ belief that an effect should be expected (e.g., O2 on sidelines) Placebo effect – subject believes that “placebo” (pill, injection, or other “treatment” that contains not active ingredient) is helping them just because they are being treated Rating Effect (Halo effect, overrater error, underrater error, central tendency error) tendency toward evaluative bias resulting from pre-existing attitudes or opinions toward whomever or whatever is being evaluated; subjects may positively or negatively evaluate something because of a previous experience with the company or product or lab solution is to select subjects without prior association (juries) or to collect additional data that might indicate direction or strength of halo effect Avis Effect (opposite of John Henry effect) effects caused by control group perceiving that they are second fiddle; subjects in control group tend to try harder Inform them of the significance of their role John Henry effect

11. Subject Expectancy: Solutions Deception Placebo Blind the subjects (double-blind) Select subjects without prior associations Inform the subjects of their roles

12. Expectancy Interactions Does the investigator respond to male subjects better than female subjects? Do the subjects respond better to one investigator better than another? Solution: randomization

13. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

14. History Outside influences that occur between the pretest and posttest in addition to the treatment Solution: Randomization Control group Length of study History is everything that happens to subjects after their selection or after the study has begun but before it has been concluded Examples of these events may be social, emotional, physiological increased conditioning - learning changes in environment (midterm of semester) Difference in group histories can become an alternative explanation May happen to a few subjects or to all subjects. If to a few subjects, you may assume random distribution. If to all, the systematic interference is problematic. The longer the time lapse between pre and post, the greater the possibility of history becoming a rival explanation. Intervening or extraneous events that occur during the study. Solution: Cannot eliminate Minimize through careful monitoring of history of each group and termination of studies in which histories are documented to have differed in relevant ways Reduce through replication of studies in which group histories are likely to differ, provided the differences in histories are not systematic from replication to replication History is everything that happens to subjects after their selection or after the study has begun but before it has been concluded Examples of these events may be social, emotional, physiological increased conditioning - learning changes in environment (midterm of semester) Difference in group histories can become an alternative explanation May happen to a few subjects or to all subjects. If to a few subjects, you may assume random distribution. If to all, the systematic interference is problematic. The longer the time lapse between pre and post, the greater the possibility of history becoming a rival explanation. Intervening or extraneous events that occur during the study. Solution: Cannot eliminate Minimize through careful monitoring of history of each group and termination of studies in which histories are documented to have differed in relevant ways Reduce through replication of studies in which group histories are likely to differ, provided the differences in histories are not systematic from replication to replication

15. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

16. Maturation Morphological changes that occur during the study that change performance or behavior Solution: Randomization Control group Length of study Morphological changes that occur during the study that change performance or behavior. Internal conditions of individual change unrelated to external events puberty, menopause growth and development social/emotional Solution: Use a control group, similar to treatment group, but doesn’t receive treatment – maturation changes will be same for both groups, so changes will be due to treatment onlyMorphological changes that occur during the study that change performance or behavior. Internal conditions of individual change unrelated to external events puberty, menopause growth and development social/emotional Solution: Use a control group, similar to treatment group, but doesn’t receive treatment – maturation changes will be same for both groups, so changes will be due to treatment only

17. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

18. Instrumentation Bias stemming from the process of measurement in the research setting Types: Measurement device influences Testing influences Changes that occur in the measurement process that affect measurement of DV measurement error of devices objectivity of observers If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement proceduresChanges that occur in the measurement process that affect measurement of DV measurement error of devices objectivity of observers If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement procedures

19. Measurement device influences Measurement error of devices Instrument sensitivity Changes that occur in the measurement process that affect measurement of DV If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement proceduresChanges that occur in the measurement process that affect measurement of DV If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement procedures

20. Measurement device influences Reliability – repeatability, consistency of instrument Validity - accuracy Objectivity – consistency of rater, measurer

21. Reliability

22. Types of Reliability

23. Reliability Assessment Pearson r correlation Reliability of scores, not tests Homogeneity of scores Report for current study conditions

24. Validity Construct Validity – adequacy of the operational definition of variables

25. Indicators of Construct Validity of a Measure Face Validity – content of measure appears to reflect the construct being measured Criterion-Oriented Validity – scores on the measure are related to a criterion predictive – scores on measure predict behavior on a criterion concurrent – people in groups known to differ on the construct score differently on measure convergent – scores on the measure are related to other measures of the same construct discriminant – scores on the measure are not related to other measures that are theoretically different Concurrent Validity – measures the relationship between a dependent measure (the measure we think is accurate) to a criterion measure (the measure that we know to be accurate).Concurrent Validity – measures the relationship between a dependent measure (the measure we think is accurate) to a criterion measure (the measure that we know to be accurate).

26. Instrument Sensitivity Ceiling & floor effects Must consider sample characteristics (age, skill level, etc.) Ceiling & floor effects IV appears to have no effect on the DV only because participants quickly reach the maximum (or minimum) performance level Ceiling & floor effects IV appears to have no effect on the DV only because participants quickly reach the maximum (or minimum) performance level

27. Measurement device influences Solution: Calibration Use of multiple observers Training of observers Establishment of reliability, validity, and objectivity of instruments Changes that occur in the measurement process that affect measurement of DV If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement proceduresChanges that occur in the measurement process that affect measurement of DV If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement procedures

28. A researcher developed a test of the trait optimism and then compared students who were judged to be “happy” versus “not happy” to see if they had different levels of optimism as expected if the Happiness Theory were credible.

29. A researcher administered the new Test of Recreation Programming Potential to graduating recreation majors and then correlated the test scores with recreation directors of programming effectiveness after one year on the job to see whether the test scores were related to job performance.

30. A researcher tested a sample of students using the Occupational Interest Test. The test was administered again six months later to see if interests were fleeting.

31. Instrumentation Bias stemming from the process of measurement in the research setting Types: Measurement device influences Testing influences Changes that occur in the measurement process that affect measurement of DV measurement error of devices objectivity of observers If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement proceduresChanges that occur in the measurement process that affect measurement of DV measurement error of devices objectivity of observers If they occur at random, or the precision is such that the errors are trivial relative to research, then there is no threat Solution: Regular calibration, use of multiple observers, training of observers establishment of reliability, validity, and objectivity of measurement procedures

32. Testing Influences Learning effect Order effect Pretest sensitization Learning effect (esp. with repeated measures) – subjects learn from taking the test the first time Order effect – order of testing may create effects (fatigue, warm-up, etc.) Pretest sensitization – experience of pretest creates an effect itself, e.g., pretest on how much they value recreation may cause subject to seek out recreational activities on their own, regardless of an intervention Solution: Randomization of order Familiarization with testing Eliminate pretest Use subjects unaware of measurement (unethical) Use Solomon 4-group Design (does not deal with RM situation) Learning effect (esp. with repeated measures) – subjects learn from taking the test the first time Order effect – order of testing may create effects (fatigue, warm-up, etc.) Pretest sensitization – experience of pretest creates an effect itself, e.g., pretest on how much they value recreation may cause subject to seek out recreational activities on their own, regardless of an intervention Solution: Randomization of order Familiarization with testing Eliminate pretest Use subjects unaware of measurement (unethical) Use Solomon 4-group Design (does not deal with RM situation)

33. Testing Influences: Solution Familiarize with testing Randomize order (counterbalancing) Eliminate pretest Use subjects unaware of measurement Use Solomon 4-group Design Randomization Control group Learning effect (esp. with repeated measures) – subjects learn from taking the test the first time Order effect – order of testing may create effects (fatigue, warm-up, etc.) Pretest sensitization – experience of pretest creates an effect itself, e.g., pretest on how much they value recreation may cause subject to seek out recreational activities on their own, regardless of an intervention Solution: Randomization of order Familiarization with testing Eliminate pretest Use subjects unaware of measurement (unethical) Use Solomon 4-group Design (does not deal with RM situation) Learning effect (esp. with repeated measures) – subjects learn from taking the test the first time Order effect – order of testing may create effects (fatigue, warm-up, etc.) Pretest sensitization – experience of pretest creates an effect itself, e.g., pretest on how much they value recreation may cause subject to seek out recreational activities on their own, regardless of an intervention Solution: Randomization of order Familiarization with testing Eliminate pretest Use subjects unaware of measurement (unethical) Use Solomon 4-group Design (does not deal with RM situation)

34. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

35. Regression Tendency of repeated measurements to shift away from extreme values toward more normal ones Ceiling/floor effect Solution: random selection, random assignment, control group Tendency of individuals within groups to become more similar after common effects. Occurs when groups are formed on the basis of an extreme score on some measure (e.g., clinical drug trial on the sickest of groups) Groups will “regress” toward the overall average EX: high skill vs. low skill, high success vs. lo success Occurs because there is always some degree of unreliability in the measuring device ceiling/floor effect law of large numbers Solution: Random assignment of groups so that vulnerability to regression effects is same for all groups Tendency of individuals within groups to become more similar after common effects. Occurs when groups are formed on the basis of an extreme score on some measure (e.g., clinical drug trial on the sickest of groups) Groups will “regress” toward the overall average EX: high skill vs. low skill, high success vs. lo success Occurs because there is always some degree of unreliability in the measuring device ceiling/floor effect law of large numbers Solution: Random assignment of groups so that vulnerability to regression effects is same for all groups

37. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

38. Mortality The differential loss of subjects from various experimental groups Solution: Large sample size Inform subjects of importance Incentives Length of study More serious problem is the systematic loss of subjects who drop out because of a common reaction to the treatment due to dropout, illness, death, boredom, fear (subject attrition) EX: randomly assign males and females to have equal number of each in experimental and control group – males drop out of experimental and females drop out of control Solution: Large sample size (if loss is not systematic) Completely and carefully inform all subjects about importance of completing study More serious problem is the systematic loss of subjects who drop out because of a common reaction to the treatment due to dropout, illness, death, boredom, fear (subject attrition) EX: randomly assign males and females to have equal number of each in experimental and control group – males drop out of experimental and females drop out of control Solution: Large sample size (if loss is not systematic) Completely and carefully inform all subjects about importance of completing study

39. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

40. Selection Errors resulting from differences in groups at beginning of study volunteer subjects intact groups Solution: Random selection Random assignment Control group Errors resulting from non-random subject selection or assignment. Groups were different to begin with because they were not randomly selected volunteer subjects intact groups Solution is randomization If study has selection bias, then it is like an ex post facto study IV is out of experimenter’s control (could not randomly manipulate or assign – subjects “self-selected” definitive conclusions can not be drawnErrors resulting from non-random subject selection or assignment. Groups were different to begin with because they were not randomly selected volunteer subjects intact groups Solution is randomization If study has selection bias, then it is like an ex post facto study IV is out of experimenter’s control (could not randomly manipulate or assign – subjects “self-selected” definitive conclusions can not be drawn

41. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

42. Intra-subject Interaction Subjects influenced by other subjects Solution: Ask subjects not to discuss study Subjects influenced by other subject’s perceptions (e.g., exit polls on East Coast) Solution: Ask subjects not to discuss study Subjects influenced by other subject’s perceptions (e.g., exit polls on East Coast) Solution: Ask subjects not to discuss study

43. Threats to Internal Validity Experimenter expectancy Subject expectancy History Maturation Instrumentation Regression Mortality Selection Intra-subject interaction Factor interaction

44. Factor Interaction The cumulative effect resulting from the combination of two or more threats. selection + maturation selection + history selection + instrumentation history + subject mortality selection + subject mortality Maturation or history effect is not the same for all groups selected. Particularly a problem when random selection is not used. Groups may differ initially in age or background, which may affect the way that they respond to treatments, e.g., they may not have the same potential for improvement.Maturation or history effect is not the same for all groups selected. Particularly a problem when random selection is not used. Groups may differ initially in age or background, which may affect the way that they respond to treatments, e.g., they may not have the same potential for improvement.

  • Login