Outline. Validity of Inference Theory of Validity Statistical Conclusion Validity Internal Validity Construct Validity – Jill External Validity – Tim Trade-offs – Tim et al Discussion. Validity of Inference. VALIDITY. The approximate truth of an inference
Validity of Inference
Theory of Validity
Statistical Conclusion Validity
Construct Validity – Jill
External Validity – Tim
Trade-offs – Tim et al
Validity of Inference
Why is it important to remember that validity is a property of a knowledge claim, not a property of the design?
Correspondence between empirical evidence and abstract inferences
Sensitive to degree of coherence between findings and theory
Pragmatic ruling out of alternative explanations
Truth is a social construction!
Followed Campbell (1957) closely in defining internal and external validity.
Internal validity: inferences about whether “the experimental treatments make a difference in this specific experimental instance.” (p. 5)
Construct validity: asked “to what populations, settings, treatment variables and measurement variables can this effect be generalized?” (p. 5)
To draw generalized causal inferences it is useful to treat the causal and generalizability aspects of the inferences separately:
How large and reliable is covariation between the presumed cause and the effect?
Is the covariation causal, or would it have been obtained without the treatment?
Which general constructs are involved in the persons (units), treatments, observations, and settings (UTOS)?
How generalizable is the locally-embedded causal relationship over varied UTOS?
These questions and inferences are often considered separately, so it is practical to have the typology reflect that
However, they are often related - and different combinations are possible (e.g., internal validity with or without construct validity)
Interesting to consider the limits of combinations (e.g., to what extent is both high internal and external validity possible?)
(sign of effect is +)
(sign of effect is -)
Can we prove that covariation between a treatment and an outcome is zero?
The validity of inferences about whether observed covariation between A (treatment) and B (outcome) reflects a causal relationship from A to B as those variables were manipulated or measured.
Is the covariation causal or would the same effect be obtained without treatment?
*GARDASIL DAILY DOUBLE*
Threats to internal validity are not necessarily independent of each other. Define two threats to internal validity and explain how they could be related / co-occur in a study.
Both concern operations (not the constructs they represent)
Statistical conclusion validity is concerned with errors in assessing covariation
Internal validity is concerned with errors in causal-reasoning
Internal validity depends substantially on statistical conclusion validity
What happens to the precision, and confidence intervals, of effect size estimates when a study has low power?
What kind of validity is threatened?
A specific instance of selection bias is also defined in SCC’s list as a separate threat to internal validity. What is it?
Confounding of treatment effects with population differences threatens _______ validity
You are a part of a research team that has been funded to tackle the adult obesity epidemic. The hypothesis is that adults receiving the intervention will have a healthier weight than adults who do not receive the intervention. You ask your boss, “How will we measure healthy weight?” To which, your boss replies, “Simple, we will ask each participant their height and weight.” You ask, “That’s it?”, and your boss replies, “Yes.”
You’re new to the team, but you really want to speak up because this is a threat to _________ validity, known as ____________________.
*HERPES DAILY DOUBLE*
Random sampling, though rarely performed in experimental designs, improves what kind of validity?
*HEPATITIS DAILY DOUBLE*
You work at High Times Community College and your coworker comes to work sharing the results of a new study. He says, “Listen to this! In a new study, students were randomly assigned to take 10, 15, or 20 units of course credit. Results show that college students who took 20 or more credits were less likely to engage in marijuana use. So to reduce the prevalence of marijuana use here at High Times CC, we have to implement a policy putting a minimum credit hour of 20 for all students!”. You, having taken H699, take a closer look at the report and see that the study was conducted at one university…Harvard. Your response to your colleague is, “Sorry, my friend, but this study most likely lacks _________ because ____________________”
You want to test out a novel approach to improving psychological distress among college students. Your technique is provided to students that come into the campus counseling center. You conduct two week follow-ups with these students and see that their self-reported levels of psychological distress has improved. You are ready to tell your boss about the success of your program when your colleague points out that your study has a threat to _______ validity known as ____________.
Secular trends pose a threat to _________validity.
You have completed an RTC in which you examined the impact of an SAT preparation course on SAT performance. You want to see if results differ for boys versus girls.
What will happen to your power if your sample is divided by gender?