Practical Meta-Analysis for the Social Sciences

1. Practical Meta-Analysis for the Social Sciences Evan J. Ringquist School of Public and Environmental Affairs Indiana University Workshop in Methods Presentation January 11, 2013 Bloomington, IN 47405

2. “Meta-Analysis is not a fad. It is rooted in the fundamental values of the scientific enterprise: replicability, quantification, causal and correlational analysis. Valuable information is needlessly scattered in individual studies. The ability of social scientists to deliver generalizable answers to basic questions of policy is too serious a concern to allow us to treat research integration lightly. The potential benefits of meta-analysis method seem enormous.” (Bangert-Drowns 1986: 398)

3. Meta-Analysis Defined • Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies. • Meta-analysis, then, provides a powerful set of tools for aggregating knowledge.

4. Number of Articles Published Each Year in the “Meta-Analysis” Topic Category as Referenced by the Social Sciences Citation Index

5. Outline of Presentation • Motivation for Using Meta-Analysis in the Social Sciences • Quick Introduction to Meta-Analysis • Meta-Regression Models: The Basics • Meta-Regression Analysis for the Social Sciences • CRVE, GEE, and if we have time, HLM

6. Section I: Motivation

7. Cumulative Knowledge and the Scientific Enterprise • A Central Goal of Science is the Accumulation and Aggregation of Knowledge. Yet the Social Sciences in General and Researchers in Public Management and Policy in Particular Have Found Cumulative Knowledge to be an Elusive Goal.

8. “Cumulative Knowledge” in Environmental Justice • “Overwhelming evidence” of significant environmental inequities (Goldman 1993; see also Mohai and Bryant 1994) • “even a reasonably generous reading of the empirical research alleging environment inequity . . . must leave room for profound skepticism” (Foreman 1998; see also Bowen 2001)

9. “Cumulative Knowledge” in Education Policy • “Extensive research has been conducted on the academic success of students enrolled in school choice program nationwide. Rigorous studies show strong gains for voucher and scholarship tax credit recipients . . .” (Alliance for School Choice, 2010) • “The reason that vouchers had subsided as a point of advocacy is because they don’t work.” (Randi Weingarten, President, American Federation of Teachers; Associated Press 2011). • “Few contemporary questions in American education have produced such wide-spread controversy as that regarding the potential of school vouchers to reduce inequality in student outcomes. . . these efforts notwithstanding, answers to the voucher question still appear uncertain.” (Cowen 2008)

10. Cumulative Knowledge in Public Management • the more knowledge we create and diffuse, “the less we know, because our research is not aggregated or accumulated into substantial bodies of knowledge” (Van Slyke, O’Leary, and Kim 2010: 290. • The first step in making Public Administration a stronger and more robust field is the need to aggregate knowledge in the sense of making it cumulative. (Rosenbloom 2010)

11. Cumulative Knowledge in Policy Studies • “political scientists have spent (almost literally) countless articles and books proposing something like ‘laws’ or theories that, taken collectively . . . have produced infinitely more confusion than clarity” (DeLeon 1998: 150).

12. Cumulative Knowledge in Policy Evaluation • Social scientists identify new and important questions having policy relevance. Initial studies examining these questions provide clear answers regarding, for example, the effectiveness of social policy interventions. Subsequent studies cast doubt on these initial conclusions, however, and with the proliferation of studies comes a proliferation of conclusions. The best efforts of social and behavioral researchers generate confusion and uncertainty rather than clarity. In the end, researchers conclude that the phenomena being studied are ‘hopelessly complex’ and move on to other questions. After several repetitions of this cycle, social and behavioral scientists themselves become cynical about their own work and express doubts about whether behavioral and social science in general is capable of generating cumulative knowledge or answers to socially important questions (Hunter and Schmidt 1996: 325-6).

13. Why Do Social Scientists Struggle with Knowledge Accumulation? • Epistemological Differences: Social Sciences ought not aim to generate generalizable “scientific” knowledge ala Chemistry. • Subject Matter Differences: Social scientists study organizations and institutions devised and populated by strategic actors, not molecules or organizms without agency. • Social Scientists Use The Wrong Tools for Aggregating Knowledge.

14. Section II: Introduction to Meta-Analysis

15. Meta-Analysis Defined (again) • Meta-analysis is a systematic, quantitative, replicable process of synthesizing numerous and sometimes conflicting results from a body of original studies. • Meta-analysis, then, provides a powerful set of tools for aggregating knowledge.

16. The Language of Meta-Analysis • Original Study: A piece of original research, published or unpublished, that aims to test a hypothesis and/or estimate a quantity of interest. • Focal Predictor: The independent variable in an original study that measures the key exogenous concept associated with the research question of interest.

17. The Language of Meta-Analysis • Effect Size: A standardized measure of the relationship between the focal predictor and the dependent variable in an original study. Effect sizes are the unit of analysis and the quantity of interest in a meta-analysis, often designated Θi. Without effect sizes there can be no meta-analysis.

18. The Language of Meta-Analysis • Effect Size Variance: A measure of the uncertainty associated with a particular effect size. • Fixed and Random Effects Models: Methods of weighting effect sizes that embody different assumptions about the effect size variance.

19. How Do We Calculate Effect Sizes? • Three Families of Effect Sizes • D-based effect sizes (standardized mean differences, common in education and psychology) • Odds-based effect sizes (e.g. log odds of an event, common in medicine) • R-based effect sizes (partial correlation coefficients, unusual in the literature but most useful for social scientists)

20. Calculating R-based Effect Sizes from Original Studies • r = √[t2 / (t2 + df)] [1] • r = √[Z2 / n] [2] • r = √[Χ21 / n] [3] • V[r] = (1-r2)2 / (n-1) [4]

21. Fisher’s Corrections for R-based Effect Sizes and Variances • Zr= 0.5 ln[(1+r) / (1-r)] [5] • V[Zr] = 1/(n-3) [6]

22. Example: Are Pollution Emissions Higher in Black Neighborhoods? • Ringquist 1997: t=3.06, df=29202 • r=.02 • Zr=.02 • V[Zr] = .000034 • Downey 1998: t=3.38, df=112 • r=.30 • Zr=.31 • V[Zr] = .0072

23. What Can We Do With Effect Sizes Statistically? • Communicate results from original research in a more meaningful fashion (not meta-analysis). • Combine (average) effect sizes to estimate the population effect size • Test null hypothesis that population effect size equals zero • Explain variation in effect sizes across original studies. We use “Meta-Regression” to account for this variation.

24. Combining Effect Sizes • We calculate average effect sizes, or our estimate of the population effect size, by calculating a weighted average of all effect sizes from original studies where the weights are inverse variances • Θbar= ΣwiΘi / Σwi [7] • Wi = 1/vi

25. Combining Effect Sizes: Fixed Effects Models • Fixed Effects models assume that effect sizes vary across original studies only due to sampling error: • Θi= Θ + ei[8], so • E[Θi]= Θ [9], and • ei ~ N(0,vi) [10]

26. Combining Effect Sizes: Fixed Effects Models • Fixed effects meta-regression assumes that effect sizes conditional upon moderator variables differ only due to sample size • Fixed effects estimates apply only to sample of original studies in hand

27. Combining Effect Sizes: Random Effects Models • Random Effects models assume that effect sizes are normally distributed random variables • Θ ~ N(µΘ, τ2) [11] • Θi = µΘ + ei [12] • E[Θi] = µΘ [13], and • ei ~ N(0,vi + τ2) [14] • Always use random effects in the social sciences!

28. Estimating the Random Effects Variance Component • Using Restricted Maximum Likelihood • Le(τ2) = -.5 * Σ [ln(vi + τ2) + ((ei2 / (vi + τ2) - .5 * ln |X’v-1X| [15] • Using Method of Moments (MOM) • tr(M) = Σvi-1 – tr [(Σvi-1 XX’)-1 (Σvi-2 XX’)][16] • Using MOM Approximation • τ2 = [SSEols / (m-k-1)] – vbar[17]

29. What Can We Do With Effect Sizes Substantively? • Estimate severity of problems • E.g., environmental inequities • E.g., effects of climate change • Measure important quantities of interest • E.g., statistical value of a life • E.g., hedonic pricing of environmental amenities

30. What Can We Do With Effect Sizes Substantively? • Program Evaluation • E.g., effectiveness of educational vouchers • E.g., effectiveness of job training programs • Theory Testing and Development • E.g., Top-down vs. bottom-up implementation • E.g., effects of negative campaign ads • E.g., Ricardian equivalence

31. Section III: Introduction to Meta-Regression

32. “exploration of the pattern of variation in effect sizes among studies is a far more important goal of meta-analysis than the construction of powerful tests of null hypotheses.” (Osenberg et al. 1999: 1105)

33. Moving Meta-Analysis to the Social Sciences • Techniques for calculating and combining effect sizes developed for synthesizing the results from experiments. Can these same techniques really be used to synthesize results from the non-experimental multivariate models common in the social sciences?

34. Three Critiques of Meta-Analysis in the Social Sciences • Parameter estimates from the general linear model are not comparable. • Effect sizes cannot be calculated from multivariate models. • Parameter estimates from different models estimate different population parameters, and therefore cannot be combined.

35. Response to Critique #1: • While regression parameters are not comparable, meta-analysis does not combine parameter estimates. Rather, meta-analysis combines effect sizes which are standardized with respect to scale. This is not a valid critique.

36. Response to Critique #2 • In fact, the same formulas that can be used to calculate r-based effect sizes from experimental studies can be used to calculate partial correlations from regression models, probit models, etc. (see Greene 1993: 180)

37. Response to Critique #2 • Example 1: Panel regression model, Y=continuous, K=33, N=307,538 • Pcorr = .0099, Zr = .0096 • Example 2:Probit model, Y=dichotomous, K=8, N=14131 • Pcorr = .1081, Zr = .1062 • Example 3: OLS regression model, Y=continuous, K=24, N=285 • Pcorr = .1619, Zr = .1565

38. Response to Critique #3 • E.g., Study 1 • Y = b0 + b1X1 + b2X2 + b3X3 + e • E.g., Study 2 • Y = b0 + b1X1 + b2X2 + b4X4 + e • E[b11]≠ E[b12], β11 ≠ β12, and Θ1 ≠ Θ2 • Therefore dependent variable Θi measures fundamentally different quantities

39. Response to Critique #3 • Critique #3 is a valid critique, and this is why calculating average effect sizes is of little value in the social sciences – these average effect sizes combine estimates of different population parameters, and therefore are not valid estimates of any useful quantity of interest. • Thankfully, we can address Critique #3 using Meta-Regression.

40. What Do We Need to Conduct a Meta-Regression? • Measure of effect size that is comparable across original studies employing different measures, models, and samples • Measure of (un)certainty or effect size variance • Moderator variables that account for variability in effect sizes

41. Introduction to Meta-Regression • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei~ N (0, vi) • Θiis the effect size • X represents moderator variables accounting for differences in effect sizes within and across studies • Scientifically Interesting Moderators (Rubin 1992) • e.g., differences attributable to target characteristics • e.g., differences attributable to program or policy design • Scientifically Uninteresting Moderators • e.g., differences attributable to estimation technique or research design

42. Number of Publications Using Phrase “Meta-Regression” in Google Scholar, 1990-2011

43. Example: Synthesizing Research on Educational Vouchers • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) • Θi is the estimated point biserial correlation (effect size Zr) between the use of an educational voucher and student standardized test scores from a particular statistical model in a particular original study.

44. Example: Educational Vouchers • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) • X1 moderator variable identifying effect sizes from models limiting sample to black children • X2moderator variable identifying effect sizes from voucher programs available to religious schools • X3moderator variable identifying effect sizes from models not using random assignment to treatment and control groups.

45. Example: Educational Vouchers • Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi) • b0 average effect of vouchers on student test scores for all students in experimental studies where vouchers use is limited to secular schools • b1 differential effect of vouchers for black students • b2 differential effect from voucher programs that include religious schools • b3 differential voucher effect from quasi-experimental studies

47. Obtaining Estimates for Meta-Regression Models • OLS Estimator of Meta-Regression Model • E[b]= β = (X’X)-1X’Θ[18] • var[b] =σ2(X’X)-1[19] • But OLS is incorrect for two reasons: • Θiis heteroskedastic(recall assumptions) • OLS weights all observations equally (unequal variances means unequal certainty)

48. Generalized Least Squares in Meta-Regression • In OLS, ee’ = σ2I • In GLS, ee’ = σ2Ω • where Omega is a diagonal matrix with proportional error variances on main diagonal • GLS estimates then are: • E[bgls] = βgls = (X’Ω-1X)-1X’Ω-1Θ[20] • var[bgls] = σ2(X’Ω-1X)-1[21]

49. Weighted Least Squares in Meta-Regression • Find a weight matrix W so that W’W = Ω-1 • Then the WLS meta-regression estimates become: • E[bwls] = βwls = (X’W’WX)-1X’W’WΘ[22] • var[bwls] = σ2 (X’W’WX)-1[23]

50. Weighted Least Squares in Meta-Regression (cont.) • The good news is that we have a handy estimate of the diagonal elements of the weight matrix W: the effect size variances (or more properly, their square roots) • The bad news is that fixed effects variance estimates are almost certainly wrong, so we need to use random effect variances.