Meta Analysis: Studying and Averaging Results to Estimate Population Parameters

1. Meta Analysis An Introduction

2. What… is… it? • A “study of studies,” i.e., averaging results across studies in a given domain to get a better estimate of population parameters (Allen & Preiss, 2007; Hunter & Schmidt, 2004). • Key ingredient = measures of effect size (usually Pearson’s r or Cohen’s d)

3. Why????????? • 1. To reduce problems associated with SAMPLING ERROR at the individual study level. • How big is this problem? According to Hunter, sampling is error is “massive” with sample sizes of N=100 or less, as we typically have in research. • Result: VERY frequent Type II errors….

4. Example • Monte Carlo study with population effect size (p) = .10, and 19 studies of sample sizes of N=30, N=68, and N=400. • How many times did the conventional p < .05 test flag (*) an r as significant? • 1/19 times, for a 95% error rate! • What about for N = 68? • 2/19, for an error rate of 89%. • For N = 400, the error rate is STILL 47%!

5. Why????????? • Hunter & Schmidt (1990) note that though Type I is typically 5%, Type II error rate for an average sample size (N = 80) with an average effect size (d = .40) and alpha level at p = .05 is… • …still about 50%! • How do we fix this? By increasing sample size in individual studies (rarely done) or through meta analysis.

6. But… it gets even worse…. • In addition to random (sampling) error, studies also suffer from systematic problems, such as: • Measurement artifacts. • Issues of design. • Choice of sample. • Anything else that makes study results different. • Fortunately, meta analysis can correct or at least account for these problems. Yay!

7. Points to Consider • Meta analysis isn’t as easy as it may seem on the surface (see next slides), but… • It provides the most accurate estimate of population parameters possible (vs. individual studies or literature reviews), and… • FYI: Natural sciences also have variability in study results and most use forms of meta analysis to deal with divergent study findings.

8. So how can I do a meta analysis? • It’s not available as an option in SPSS….  • It seems like many meta analysts do the calculations by hand(!), though there is some software available for it, e.g., • Jack Hunter’s free DOS programs for “bare bones” meta analysis and other corrections he advocates for. • Commercially available software like CMA.

9. Meta Analysis Example • Sherry (2001) estimated the effect size of violent video games on aggression through meta analysis. Steps included: • 1. Study selection and coding (p. 415-416) • Exhaustive lit search done for studies on video game violence and aggression, from 1975-2000 • Of 900 initial returns, 25 studies were identified from which an effect size could be estimated. • Relevant info was recorded on coding sheets.

10. Meta Analysis Example • 2. Effect sizes (r) estimated (p. 417-418) • Many had to be calculated from other stats. • Nonsignificant findings also had to be dealt with. • 3. Overall effect size estimated (p. 418-419) • Mean effect size (weighted by sample size) and variance from all studies calculated. • Sampling error residual variance accounted for. • Variance from moderating variables accounted for. • See Table 1 (p. 420) for a typical summary.

11. Results Highlights • Table 2 shows mean effect sizes and residual variance (p. 421) • High probability of moderating variables indicated. • Methodological variables include survey vs. experiment and type of outcome measure. • Theoretical variables include age, type of game violence, and length of game exposure. • Table 3 shows how the theoretical variables correlate with effect size.

12. Results Highlights • MULTIVARIATE STAT ALERT! • These variables were entered into a multiple regression equation with effect size as the DV and moderators as IVs. • Sherry’s reason? “To control for the effect of moderators on each other (e.g., suppression).” • Check out the results in first full paragraph on p. 422.

13. What did we learn? • Converting to d, overall effect size of games on aggression is .30, smaller than that found for television of .65 (Paik and Comstock, 1994). • More recent games have a larger effect size. • Player age also positively related to effect size. • Effect size negatively related to playing time, however. • Results used for theoretical advancement.

14. The Latest Meta Analysis Piece • Theoretical assessment of evidence (2007): • 1. No support for social learning theory. • 2. Little support for catharsis, though not studied properly. Time finding points to it, as does declining violence at macro level. • 3. Arousal (excitation transfer) and priming supported by available evidence. • New model: PRIMED AROUSAL.

15. You Tube BREAK • How prevalent is META ANALYSIS on YouTube? • Not very…. • BONUS VIDEO tangentially related to meta analysis and the next slide: Powerthirst 2: Re-Domination

16. Other Recent Examples • Paul et al. (2007) Third Person Effect study—32 studies (N = 45,729) indicate a substantial effect size of r = .50. Q: Moderators? • A: Message, sampling, and respondent type. • Other topics in Preiss et al. (2007) include effects of agenda setting, sexually explicit media, frightening media, music, health campaigns, spiral of silence, and more….

17. Final Considerations • Shows importance of “replication, replication, REPLICATION” (Hunter 2001) in science. • Some limitations, however…. • Have to have a number of studies in a given domain before a meta analysis is worthwhile; otherwise, a literature review should suffice (Pfau, 2007). • Q: Are there any areas you think are ripe for a meta analysis?

18. DO IT! • Questions, comments, suggestions? • Thank You.