1 / 40

# Outline - PowerPoint PPT Presentation

Outline. Validity of Inference Theory of Validity Statistical Conclusion Validity Internal Validity Construct Validity – Jill External Validity – Tim Trade-offs – Tim et al Discussion. Validity of Inference. VALIDITY. The approximate truth of an inference

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## PowerPoint Slideshow about ' Outline' - winka

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Validity of Inference

Theory of Validity

Statistical Conclusion Validity

Internal Validity

Construct Validity – Jill

External Validity – Tim

Discussion

### Validity of Inference

• The approximate truth of an inference

• Judgment about the extent to which relevant evidence supports the inference as being true

• Always entails fallible human judgments

• Evidence comes from both empirical findings and their consistency with past findings and theories

• Validity judgments are never absolute

• No certainty that inferences are true or that all possible alternatives have been falsified

• Validity is a property of inferences

• not of designs or methods

• Even using a randomized experiment does not guarantee a valid causal inference

• Could be “broken” by

• Differential attrition

• Low statistical power

• Improper statistical analysis

• Sampling error

Why is it important to remember that validity is a property of a knowledge claim, not a property of the design?

Three Theories of Truth of a knowledge claim, not a property of the design?

• Correspondence theory

• A knowledge claim is true if it corresponds to the world – e.g., see it raining

• Coherence theory

• A claim is true if it belongs to a coherent set of claims

• Pragmatism

• A claim is true if it is useful to believe it

• Philosophers do not agree on which theory of truth is correct – and for us it doesn’t matter!

• Science uses them all to approximate the truth

The of a knowledge claim, not a property of the design?Theory of Validity is pragmatic and uses them all

Correspondence between empirical evidence and abstract inferences

Sensitive to degree of coherence between findings and theory

Pragmatic ruling out of alternative explanations

Truth is a social construction!

Campbell & Stanley, 1963 of a knowledge claim, not a property of the design?

Followed Campbell (1957) closely in defining internal and external validity.

Internal validity: inferences about whether “the experimental treatments make a difference in this specific experimental instance.” (p. 5)

Construct validity: asked “to what populations, settings, treatment variables and measurement variables can this effect be generalized?” (p. 5)

Cook & Campbell (1979) Expanded Typology of Validity of a knowledge claim, not a property of the design?

To draw generalized causal inferences it is useful to treat the causal and generalizability aspects of the inferences separately:

• Statistical conclusion Validity

• Internal Validity

• Construct Validity

• External Validity

Corresponds to 4 Questions of a knowledge claim, not a property of the design?

How large and reliable is covariation between the presumed cause and the effect?

Is the covariation causal, or would it have been obtained without the treatment?

Which general constructs are involved in the persons (units), treatments, observations, and settings (UTOS)?

How generalizable is the locally-embedded causal relationship over varied UTOS?

These questions and inferences are often considered separately, so it is practical to have the typology reflect that

However, they are often related - and different combinations are possible (e.g., internal validity with or without construct validity)

Interesting to consider the limits of combinations (e.g., to what extent is both high internal and external validity possible?)

Threats To Validity separately, so it is practical to have the typology reflect that

• Are specific reasons why we can be partly or completely wrong in our inferences

• About covariation, causation, constructs or variations across UTOS

• It is useful to anticipate criticisms of inferences by considering the types of limitations encountered by past research.

• Heuristics, such as a list of potential threats, allow us to account for threats in the design or by including measures of anticipated threats.

3 Critical Questions about Threats separately, so it is practical to have the typology reflect that

• For any particular experiment and finding:

• How would the threat apply in this case?

• Is there other evidence that the threat is plausible rather than just possible?

• Does the threat operate in the same direction as the observed effect (so that it could partially or totally explain it)?

• But “ruling out” threats is a falsification enterprise, so is always limited

Statistical Conclusion separately, so it is practical to have the typology reflect thatValidity

• The validity of inferences about the covariation or correlation between the treatment and the outcome

• How large and reliable is the covariation?

• Whether the variables covary or not

• How strongly they covary (SCC, p. 42)

Testing covariation separately, so it is practical to have the typology reflect that

• Null hypothesis significance testing (NHST)

• Common misunderstandings of p value

• NHST tells little about effect size

• Effect size bound by confidence intervals

• An alternative approach

• SCC recommend these along with exact p of type I error

• Classical separately, so it is practical to have the typology reflect thatInterpretation of p value

• In the classic interpretation, exact Type I probability levels tell us the probability that the results that were observed in the experiment could have been obtained by chance from a population in which the null hypothesis is true (Cohen, 1994 as cited in SCC, p. 44).

• “Perhaps not the most interesting hypothesis” (SCC)

Alternative separately, so it is practical to have the typology reflect thatInterpretation of p value

• p value (probability level) signifies the confidence we can have in deciding among the following claims:

• 1) Treatment A did better than treatment B

(sign of effect is +)

• 2) Treatment B did better than treatment A

(sign of effect is -)

• 3) The sign is uncertain (P > .05 signifies 3, “too close to call”)

Incorrect statistical conclusions separately, so it is practical to have the typology reflect that(SCC, p.42)

• 1) Whether the variables covary

• Type I error (claim of a difference when there is none)

• Type II error (conclude that there is no effect when in fact there is one)

• 2) How strongly they covary

• Overestimate magnitude of covariation (and confidence in estimate of magnitude)

• Underestimate magnitude of covariation (and confidence in estimate of magnitude)

• Threats to Statistical Conclusion separately, so it is practical to have the typology reflect thatValidity

• Low statistical power

• See Table 2.3 (pp. 46-7) for methods to increase power

• Violated assumptions of the test statistics

• Fishing and the error rate problem

• Unreliability of measures

• Always attenuates bivariate relationships

• Restriction of range – floor and ceiling effects

• Unreliability of treatment implementation

• Extraneous variance in experimental setting

• Heterogeneity of respondents (units)

• Inaccurate effect size estimation

To support the causal inferences, three things must be established (p. 53):

• 1) A precedes B in time (use design)

• 2) A covaries with B (use statistics)

• 3) No other explanation for the relationship is plausible (use design if possible)

I established (p. 53):nternal Validity

• The ability to infer with confidence that an independent variable has produced the observed differences in the dependent variable (Singleton & Straits, 2005, p. 188)

• Isolating the independent variable

• Controlling confounds

• Validity: the approximate truth of an inference (SCC, p. 34)

Internal validity established (p. 53):

The validity of inferences about whether observed covariation between A (treatment) and B (outcome) reflects a causal relationship from A to B as those variables were manipulated or measured.

Is the covariation causal or would the same effect be obtained without treatment?

Internal validity established (p. 53):

• “Local Molar Causal Validity”

• Local: generalizability is zero, limited to UTOS

• Molar: treatments are a complex package

• Causal: restricted to claims that “A caused B”

• “One of the things that's most difficult to grasp about internal validity is that it is only relevant to the specific study in question” (Trochim, 2006).

Threats to Internal established (p. 53):Validity

• Each threat signifies a distinct class of extraneous other possible causes (p. 55)

• Ambiguous temporal precedence

• Selection bias

• History

• Maturation

• Statistical regression

• Attrition (a special case of selection bias)

• Testing effects

• Instrumentation

• Additive/Interactive effects of these threats

*GARDASIL DAILY DOUBLE* established (p. 53):

Threats to internal validity are not necessarily independent of each other. Define two threats to internal validity and explain how they could be related / co-occur in a study.

Randomization Controls Most Threats to Internal Validity established (p. 53):

• Indeed, all except

• Differential attrition

• Differential testing

Relating Statistical established (p. 53):and Internal Validity

Both concern operations (not the constructs they represent)

Statistical conclusion validity is concerned with errors in assessing covariation

Internal validity is concerned with errors in causal-reasoning

Internal validity depends substantially on statistical conclusion validity

Jill and Tim established (p. 53):

• Jill

• Construct Validity

• Tim

• External Validity

• Evaluators discuss external validity much less than internal validity

• Some idea of disagreements in the field(s)

• Threats to validity overlap

• E.g, Attrition is listed as a threat to internal validity. But because sample size drops, it can threaten power (statistical conclusion validity), may require changing how we describe who is and is not in the study (construct validity), and may raise questions about whether the intervention would have the same effect in those who dropped out (external validity).

Threats to validity established (p. 53):Discussion questions

What happens to the precision, and confidence intervals, of effect size estimates when a study has low power?

What kind of validity is threatened?

A specific instance of selection bias is also defined in SCC effect size estimates when a study has low power? ’s list as a separate threat to internal validity. What is it?

You are a part of a research team that has been funded to tackle the adult obesity epidemic. The hypothesis is that adults receiving the intervention will have a healthier weight than adults who do not receive the intervention. You ask your boss, “How will we measure healthy weight?” To which, your boss replies, “Simple, we will ask each participant their height and weight.” You ask, “That’s it?”, and your boss replies, “Yes.”

You’re new to the team, but you really want to speak up because this is a threat to _________ validity, known as ____________________.

Random sampling, though rarely performed in experimental designs, improves what kind of validity?