Evaluation Research and Meta-analysis

1. Evaluation Research and Meta-analysis

2. Terms • Evolutionary epistemology • Evidence-based practice • Systems thinking • Dynamical systems approaches • Evaluation research

3. Issues with evaluation research • What questions are asked? • What methods are used? • What unique issues emerge?

4. Types of evaluation • Formative • Needs assessment • Evaluability assessment • Structured conceptualization • Implementation evaluation • Process evaluation • Summative • Outcome evaluation • Impact evaluation • Cost-benefit analysis • Secondary analysis • Meta-analysis

5. Methods used for different ?s • What is the scope of the problem? • How big is the problem? • How should we deliver the program? • How well did we deliver it? • What type of evaluation can we do? • Was the program effective? • What parts of the program work? • Should we continue the program?

6. Evidence based medicine (Sackett et al.) • Convert problem into question • Find evidence • Evaluate validity, impact, applicability • Integrate patient experience and clinical judgment • Review evaluation

7. Significance • What is the problem with just using p-levels to determine whether one variable has an effect on another? • Don’t EVER just give p-range! • Significance test = effect size x study size • What is the difference between statistical, practical, and clinical significance? • Sample results: • For boys, r (87) = .31, p = .03 • For girls, r (98) = .24, p = .14

8. What should you report? • 2 group comparison—treatment vs. control on anxiety symptoms • 3 group comparison—positive prime vs. negative prime vs. no prime on number of problems solved • 2 continuous variables—relationship between neuroticism and goal directedness • 3 continuous variables—anxiety as a function of self-esteem and authoritarian parenting

9. Narrative vs. quantitative reviews • When was the first meta-analysis? • When was the term first used? • What are the advantages of quant reviews? • What are particular critiques of them? • What are the three basic principles to guide meta-analysis?

10. Steps to meta-analysis

11. 1. define your variables/question • 1 df contrasts • What is a contrast?

12. 2. Collect studies systematically • Where do you find studies? • File drawer problem • Rosenthal’s fail-safe N • # studies needed at p < .05= (K/2.706) (K(mean Z squared) = 2.706) • Z = Z for that level of p • K = number of studies in meta-analysis • Funnel plot • Rank correlation test for pub bias • What can you do if publication bias is a problem? • Trim and fill • Weight studies

13. 3. Calculate effect sizes • When should you report which effect size, and what do they mean? • What does the sign mean on an effect size? • What are small, medium, and large effects? • How can you convert from one to another? • r or d? • http://www.soph.uab.edu/Statgenetics/People/MBeasley/Courses/EffectSizeConversion.pdf

14. Families of effect sizes • 2 group comparisons (difference between the means) • Cohen’s d • Hedge’s g • Glass’s d • Delta ∆ • Continuous or multi-group (proportion of variability) • Eta squared η2 • Partial eta-squared ηp2 • Generalized eta-squared η G2 • r, fisher’s z, R2, adjusted R2 • ω2 and its parts

15. Nonparametric effect sizes • Nonnormal data: convert z to r or d • Categorical data: • Rho • Cramer’s V • Goodman-Kruskal’s Lambda • Interpretation of effect sizes • PS • U • Binomial effect size display

16. BESD • Binomial effect size display • Relative risk • Odds ratio • Risk difference

17. 4. Look at heterogeneity of effect sizes • Chi-square test • Standard deviations of effect sizes • What are common moderators you might test? How would you do that?

18. 5. Combine effect sizes • When should you do fixed vs. random effects? • Should you weight effect sizes, and if so, on what? • How can you deal with dependent effect sizes? • Hunter and Schmidt method vs. Hedges et al. method • Credibility intervals vs. confidence intervals

19. 6. Calculate confidence intervals/ 7. Look for moderators • What are common moderators you might test? • How do you compare moderators?

20. “Meta-analysis” • Comparing and combining effect sizes on a smaller level—when might you want to do this? • How would you do it? • Average within-cell r’s with fisher z transforms • To compare independentr’s: Z = z1-z2/sqrt ((1/n-3) + (1/n-3)) • To combine independentr’s: z = z1+z2/2

21. Write-up • Inclusion criteria, search, what effect size • Which m-a tech and why • Stem and leaf plots of effect sizes (and maybe mods) • Forest plots, bean plots • Stats on variability of effect sizes, estimate of pop effect size and confidence intervals • Publication bias analyses

22. What does the book author • Mean by an “evaluation culture”? • Is it a good thing?

23. Post spring break • Readings on analyses (reading on reserve in library) • Quant article critique is separate from thought paper, due Wed. (look for questions at end of syllabus) • One more week then rough drafts due