Validity

1. Validity RMS – May 28, 2014

2. Measurement Reliability • The extent to which a measurement gives results that are consistent

3. Measurement Validity • The degree to which an assessment measures what it is supposed to measure

4. Construct validity • The extent to which an operationalization of a construct (i.e., practical tests developed from a theory) actually measure what the theory says they do • Does an IQ questionnaire actually measure intelligence?

5. Content Validity • Systematic examination of the test content to determine whether it covers a representative sample of the behaviour domain to be measured • Does the IQ questionnaire have items covering all areas of intelligence known in the literature?

6. Face validity • An estimate of whether a test appears to measure a certain criterion (no guarantee that it actually does!) • Related to content validity, but you don’t have to know all the theories to make the judgment

7. Criterion validity • Are the test and a criterion variable(s) representative of the same theoretical construct related • Compares the test with other measures already held to be valid • IQ tests validated against measures of academic performance (the criterion)

8. Assignment 2 – Variables & Measurement 1. Recap your research question and hypotheses 2. Identify the independent and dependent variables of interest for each hypothesis. For each variable: a. How could you measure the variable (i.e., what is its operational definition)? b. Should it be continuous or discrete? c. What is its scaling type? d. What descriptive statistics are appropriate? e. How would you estimate reliability and validity of measurement?

9. Questions about assignment 2?

10. Experimental validity

11. Validity • Experimental validity – the soundness of the experimental design • Not the same as measurement validity (the goodness of the operational definition) • Internal validity • Construct validity • External validity

12. Internal Validity • Concerns the logic of the relationship between the independent and dependent variables • It is the extent to which a study provides evidence of a cause-effect relationship between the IV and the DV

13. Confounding variables • Error that occurs when the effects of two variables in an experiment cannot be separated – results in a confused interpretation of the results • Example • Group A (IV = 0) • Group B (IV = 1) • Behaviour observed • Want to say that changes in behaviour are due to IV

14. How do you know what could be confounding? • Need to make judgments as you design the experiment • Be particularly careful with subject variables

15. Construct Validity • Extent to which the results support the theory behind the research • Would another theory predict the same experimental results? • Hypotheses cannot be tested in a vacuum • The conditions of a study constitute auxilliary hypotheses that must also be true so that you can test the main hypothesis

16. Example • H1: Anxiety is conducive to learning • Participants selected on basis of whether they bit their fingernails (sign of anxiety) • Observe how fast they can learn to write by holding a pencil in their toes (a learning task) • Did not just test impact of anxiety of learning • Tested that fingernail biting is a measure of anxiety (H2) • Tested that writing with toes is a good learning task (H3) • If either is false, you could have negative results even if the main hypothesis is true

17. Similarities Construct Validity Internal Validity Try to rule out alternative variables as potential causes of the behaviour of interest May be able to redesign the study to control for the source of confounding • Try to rule out other possible theoretical explanations of the results • Must design a new study to help you choose between competing theoretical explanations

18. External Validity • How well the findings of an experiment generalize to other situations or populations • Strictly speaking results are only valid for other identical situations • Can be hard to know which situational variables are important • Ecological validity is related • Extent to which an experimental situation mimics a real-world situation

19. Statistical Validity • Extent to which data are shown to be the result of a cause-effect relationship rather than an accident • Does the relationship exist or was it caused by pure chance • Similar to internal validity • Notion of power (is n big enough?) • Is measure accurate? • Is the statistical test appropriate? • 3 kinds of lies: lies, damn lies, and statistics • Statistical test establishes that an outcome has a certain low probability of happening by chance alone • No guarantee that it’s not a random error in sampling or measurement

22. So we did the green study again and got no link. It was probably a – RESEARCH CONFLICTED ON GREEN JELLY BEAN/ACME LINK; MORE STUDY RECOMMENDED - xkcd.com/882/

23. Threats to Internal Validity - 1 • Events outside the laboratory (history): • Can occur when different experimental conditions are presented to subjects at different times • Example: Feelings of depression (failure condition on Monday, success condition on Wednesday) • What if Monday was rainy and Wednesday was sunny? • Maturation: • A source of error in an experiment related to the amount of time between measures • Subjects may change between conditions because of naturally occurring processes • Aging doesn’t just occur with people!

24. Threats to Internal Validity - 2 • Effects of Testing: • Changes caused by testing procedure (not processes unrelated to the experiment) • Learn how to do the task, learn the style of a test • Regression Effect: • tendency of subjects with extreme scores on a 1st measure to score closer to the mean on a second testing • Not a perfect correlation between 2 variables • SAT and GPA or repeated SAT • Arises when there is an error associated with the measurement of the variables (e.g., students know some answers and make lucky/unlucky guesses at the rest) • Random error – that part of the value of a variable that can be attributed to chance

25. Threats to Internal Validity - 3 • Mortality: • The dropping out of some subjects before an experiment is completed • Selective subject loss • Can introduce bias depending on the reason they are dropping out • Selection: • Any bias in selecting groups can undermine internal validity

26. Threats to Construct Validity – 1 • Difficult as there are an indefinite number of theories that may account for a given relationship • Strategy: ask whether alternative theoretical explanations of the data are less plausible than the theory believed to be supported by the research

27. Threats to Construct Validity - 2 • Loose connection to theory and method • The anxiety/learning example • Often research suffers from poor operational definition of theoretical constructs • Ambiguous effect of IV • Experimenter can control all reasonable confound variables • Participant may compromise result by seeing the situation differently than the experimenter

28. Human-subject research • Good subject tendency: • Participants want to help the research achieve their goals (and may not understand the goals) • Evaluation apprehension: • Tendency of participants to alter their behaviour to appear as socially desirable as possible

29. Threats To External Validity • Other subjects • Are the participants truly representative of the sample to which you are trying to generalize? • Other times • Would the same experiment conducted at another time produce the same results? • Caution: the web is rapidly changing • Other settings • Lab vs field • Structured vs unstructured environments

30. Reading: Exp. Challenges in Cyber Security • What’s the research problem? • What was their approach? • What types of validity do they discuss? • Hypotheses? • IV, DV • Measurement validity • Internal validity • Construct validity • External validity