External validity: to what populations do our study results apply ?

External validity: to what populations do our study results apply? Epidemiology matters: a new introduction to methodological foundations Chapter 12

Seven steps • Define the population of interest • Conceptualize and create measures of exposures and health indicators • Take a sample of the population • Estimate measures of association between exposures and health indicators of interest • Rigorously evaluate whether the association observed suggests a causal association • Assess the evidence for causes working together • Assess the extent to which the result matters, is externally valid, to other populations Epidemiology Matters – Chapter 1

Generalizability or external validity refers to our capacity to generalize our results beyond our study sample Epidemiology Matters – Chapter 12

Question: How can we assess the extent to which results of a study are applicable in populations outside of the underlying population base of particular study? Answer: Think through characteristics of population of interest and determine how robust study findings might be across populations with similar or different characteristics. Epidemiology Matters – Chapter 12

Validity, four stages • Introduction to external validity • Prevalence of component causes • Causation and study design • External versus internal validity • Randomized control trials • Representative samples • Summary Epidemiology Matters – Chapter 12

Four types of validity • Measurement validity • Statistical conclusion validity • Internal validity • External validity Epidemiology Matters – Chapter 12

1. Measurement validity • An association cannot be valid beyond the study sample unless it is valid within the study sample • Accuracy and precision are key measurements • Have we measured what we wanted to measure? Epidemiology Matters – Chapter 12

2. Statistical conclusion validity • Is the association observed due to chance? • We assess this via confidence intervals around estimates of association to describe role of sampling variability • We aim to rule out the potential that our results arose by chance in the sampling process from an underlying population of interest Epidemiology Matters – Chapter 12

3. Internal validity • Assessment of non-comparability between exposed and non-exposed in any study Epidemiology Matters – Chapter 12

4. External validity • Explore external validity after assessing and ensuring • Measurement validity • Statistical conclusion validity • Internal validity Epidemiology Matters – Chapter 12

External validity External validity: the applicability of study findings beyond the study sample In an epidemiologic study we • Identify a population of interest • Sample from population – random or purposive • Conduct study • Sample result should reflect underlying association in population of interest Therefore, identifying population of interest is central to exploring external validity once we have our findings Epidemiology Matters – Chapter 12

Prevalence of component causes To understand external validity we must understand the prevalence and distribution of component causes across populations. Epidemiology Matters – Chapter 12

Prevalence of component causes, example Question: Does exposure to ambient air pollution cause lung cancer? • Component cause A: Ambient air pollution and smoking; therefore, smoking will cause lung cancer only among individuals exposed to ambient air pollution • Component cause B: Genetics Epidemiology Matters – Chapter 12

Exposed genetic, diseased Exposed genetic, diseased, exposed air pollution, smoker Non-diseased Diseased Non-exposed Exposed air pollution Smoker Exposed air pollution, diseased, smoker Epidemiology Matters – Chapter 12

Prevalence of component causesexample lung cancer Population 1 Black = exposed to air pollution Dots = genetically determined Hat = smoker Exposed to air pollution and genetic (regardless of disease status and smoking status) 2 Un-exposed to air pollution and exposed to genetic (regardless of disease status and smoking status) 2 Exposed to air pollution and smoker (regardless of disease status and genetic status) 6 Un-exposed to air pollution and smoker (regardless of disease status and genetic status) 6 Epidemiology Matters – Chapter 12

Prevalence of component causesexample lung cancer Population 2 Black = exposed to air pollution Dots = genetically determined Hat = smoker Exposed to air pollution and genetic (regardless of disease status and smoking status) 2 Un-exposed to air pollution and exposed to genetic (regardless of disease status and smoking status) 2 Exposed to air pollution and smoker (regardless of disease status and genetic status) 3 Un-exposed to air pollution and smoker (regardless of disease status and genetic status) 3 Epidemiology Matters – Chapter 12

Prevalence of component causesexample lung cancer risk difference Population 1 Population 2 Black = exposed air pollution Dots = genetically determined Hat = smoker Exposed diseased = 3 = 30% risk Unexposed diseased = 2 = 20% risk Risk difference: 30% – 20% = 10% Interpretation: 10 cases of lung cancer are associated with ambient air pollution per 100 exposed Exposed diseased = 6 = 60% risk Unexposed diseased = 2 = 20% risk Risk difference: 60% – 20% = 40% Interpretation: 40 cases of lung cancer are associated with ambient air pollution per 100 exposed Epidemiology Matters – Chapter 12

Prevalence of component causesexample lung cancer interpretation • Two studies asked the same question • Both are internally valid studies because the exposed and unexposed are comparable on genetic determinism • Population 1: the causal effect is a risk difference of 40% • Population 2 : the causal effect is a risk difference of 10% Why do these two causal effects differ? • Prevalence of people exposed to ambient air pollution is the same in both studies • Prevalence of genetic determinism is same in both studies • The reason that these two causal effects diverge is the different prevalence of smoking between the two populations Epidemiology Matters – Chapter 12

Prevalence of component causesexample lung cancer interpretation • When two causes interact the measure of association for the effect of one cause on the outcome will differ across levels of the second cause • Air pollution example • Ambient air pollution and smoking are causal partners within the same sufficient cause • Prevalence of one of them (smoking) influences the causal effect of the other (ambient air pollution) on the outcome (lung cancer) • We would therefore expect the causal effect of ambient air pollution on lung cancer to differ across population where prevalence of smoking also differs • Therefore, there is no one causal effect for all populations; the causal effect is dependent on prevalence of component causes in each population • Therefore, the result from one study will be externally valid to populations in which the distribution of component causes of exposure is similar to the study sample Epidemiology Matters – Chapter 12

Causation and study design • The magnitude of an association will be applicable beyond our study to the extent that the distribution of causal partners of exposure is similar in the population • If we want to identify a cause of disease, should it be a cause absolutely and in all types of populations? Epidemiology Matters – Chapter 12

Causation and study design • Cause: a factor that was necessary for that disease to occur in an individual at that time; most causes are insufficient and unnecessary in isolation • Causal effect: epidemiology studies populations; therefore we focus on the effect of causes • We document an association between those who embody cause (exposed) and those who do not (unexposed); this is context specific and dependent on prevalence of component causes • Therefore, understanding a cause in context of causal partners is central to theory, design, and analysis Epidemiology Matters – Chapter 12

External vs. internal validity • Internal validity is a prerequisite to external validity • To achieve internal validity we need to design a study with narrow population of interest and minimize non-comparability • The resulting sample may not reflect broader swath of population beyond underlying population of interest • The more narrow a sample becomes - due to strict inclusion and exclusion criteria for internal validity - the less external validity it may have if causal partners of exposure have differing prevalence in study compared with other populations Epidemiology Matters – Chapter 12

External vs. internal validity • Balancing external and internal validity is a a trade off • To build a scientific argument for causal effect of exposure on outcome, we select study design and assess internal validity of causal question • After causal effect of exposure is established in narrow population, we expand the causal question to ask • How often? • Among whom? • Under what conditions? Epidemiology Matters – Chapter 12

An example, RCT Question: Does weight-loss drug reduce obesity among school-age children? Study details: • Recruit children for randomized drug trial with body mass index 25< (BMI) < 40 • Exclude children with diabetes • Parents must be fully participatory (monitoring children’s drug regime and attend study clinic once per week) • Baseline survey and monthly measurements • Children randomized to receive weight loss drug or placebo • Follow-up over two years Study results: • Mean BMI among drug group declines 31.5 to 26.7 (4.8 points BMI) • Mean BMI among placebo group declines from 31.4 to 28.5 (2.9 points BMI) • Reduction of 1.9 (95% CI 0.9 – 2.9) more points of BMI in drug group than placebo group Conclusion: Weight-loss drug reduced obesity among school-age children Epidemiology Matters – Chapter 12

An example, RCT Questions to ask about external validity • Are these results externally valid to a broader population all overweight children in Farrlandia? • What about overweight children in other places? • What information would we need to know in order to inform this issue? Epidemiology Matters – Chapter 12

An example, RCT Are we confident of a causal effect in the study? • Can only be externally valid if internally valid • Good reason to conclude that results obtained are approximation of causal effect of drug for population Consider characteristics of population from which participants were drawn • Good adherers to study protocol • Diabetes-free • Actively participating parents • There is evidence that drug is effective in reducing BMI Consider characteristics of larger population to assess external validity • Does action of drug interact with other factors? • Do other factors have a different prevalence in general the general population of overweight children who may be prescribed the drug?

Example,representative sample Question: Do sales tax on sugar-sweetened beverages reduce obesity among children aged 7 to 13? Study details: • Enumerate all school-age children in Farrlandia • Take random sample of 1,000 eligible Farrlandianswho are between age 7 to 13 and live in Farrlandia • Measure BMI before tax goes into effect • Measure BMI after tax across a two-year period Studyresults: • Mean BMI school-age children prior to the tax = 26.7 (95% C.I. 24.2-29.3) • Mean BMI of school-age children was 24.3 (95% C.I. 23.7-24.9) two years after tax Conclusion: Tax lowered mean BMI among school-age children Epidemiology Matters – Chapter 12

Example, representative sample Snowtownis considering a similar tax • Are Farrlandianresults externally valid to Snowtown? • What information would we need to know about Farrlandians and Snowtownians? Epidemiology Matters – Chapter 12

Example, representative sample Are we confident of a causal effect in the study? • Internal validity: If school lunches changed to healthier offerings during study period we would not make a causal claim that tax reduced BMI What are potential causal partners of soda tax? • External validity • Soda availability is component cause Is the distribution of causal partners similar across populations? • Soda is plentiful in Farrlandia • Soda is hard to find in Snowtown Epidemiology Matters – Chapter 12

Seven steps • Define the population of interest • Conceptualize and create measures of exposures and health indicators • Take a sample of the population • Estimate measures of association between exposures and health indicators of interest • Rigorously evaluate whether the association observed suggests a causal association • Assess the evidence for causes working together • Assess the extent to which the result matters, is externally valid, to other populations Epidemiology Matters – Chapter 1

epidemiologymatters.org Epidemiology Matters – Chapter 1

External validity: to what populations do our study results apply ?