Sessions 1.2-1.3: Effect Size Calculation. Funded through the ESRC’s Researcher Development Initiative. Meta-analysis. Department of Education, University of Oxford. Sessions 1.2-1.3: Effect Size Calculation. 2. Effect size calculation. The effect size makes meta-analysis possible
An Image/Link below is provided (as is) to download presentation
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.Content is provided to you AS IS for your information and personal use only.Download presentation by click this link.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.During download, if you can't get a presentation, the file might be deleted by the publisher.
Effect size calculation • The effect size makes meta-analysis possible • It is based on the “dependent variable” (i.e., the outcome) • It standardizes findings across studies such that they can be directly compared • Any standardized index can be an “effect size” (e.g., standardized mean difference, correlation coefficient, odds-ratio), but must • be comparable across studies (standardization) • represent magnitude & direction of the relationship • be independent of sample size • Different studies in same meta-analysis can be based on different statistics, but have to transform each to a standardized effect size that is comparable across different studies
Simulate ds on homemade calculator(ES.xls) • Change direction of effects • Change Ns (equal or same?) • Change SDs XLS ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 6 6
= Control = Treatment Effect size as proportion in the Treatment group doing better than the average Control group person 69% of T above 79% of T above 57% of T above 7
= Control = Treatment Effect size as proportion of success in the Treatment versus Control group (Binomial Effect Size Display = BESD): Success: 55% of T, 45% of C Success: 62% of T, 38% of C Success: 68% of T, 32% of C 8
Long focus on significance level (safe-guarding against Type I (a) error) – today focus on practical and meaningful significance. Cohen, J. (1994). The earth is round (p < .05), American Psychologist, 49, 997–1003. Why effect size? 9
Power: “Finding what is out there” Type II (b) error “not finding what is out there” Power (1 – b): the probability of rejecting a false H0 hypothesis Power of .80 or .90 in primary research Power and effect size 11
How meaningful is a “small” effect size? • A small effect size changed the course of an RCT in 1987: placebo group participants were given aspirin instead (see Rosenthal, 1994, p. 242) XLS 13
Effect sizes Within the one meta-analysis, can include studies based on any combination of statistical analysis (e.g., t-tests, ANOVA, correlation, odds-ratio, chi-square, etc). The “art” of meta-analysis is how to compute effect sizes based on non-standard designs and studies that do not supply complete data (see Lipsey&Wilson_AppB.pdf). Convert all effect sizes into a common metric based on the “natural” metric given research in the area. E.g. d, r, OR ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 14
Effect size calculation Standardized mean difference Group contrast research Treatment groups Naturally occurring groups Inherently continuous construct Correlation coefficient Association between inherently continuous constructs Odds-ratio Group contrast research Treatment or naturally occurring groups Inherently dichotomous construct Regression coefficients and other multivariate effects Requires access to covariance-variance (correlation) matrices for each included study 15
Calculating ds (1) Means and standard deviations Almost all test statistics can be transformed into an standardized effect size “d” Correlations d P-values F-statistics t-statistics “other” test statistics 16 ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 16
Cohen’s d Hedge’s g Glass’s D Various contrast effect sizes ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 18
Calculating d (1) using Ms, SDs and ns Remember to code treatment effect in positive direction! ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 19
T-test table df = (n1 + ns –2) Sometimes authors only report e.g., p<.01 (n = 22). If so, use a conservative approach to reading the t-test table. NOTE: When p = n.s. some researchers code d = 0 in data base 26 26
Use all available tools for calculating the following 5 effect sizes • ES 6: MT = 21, MC = 20, nT = 60, nC = 60, t = .55 • ES 7: MT = 103.5, MC = 100, SDT = 22.0, SDC = 18.5, nT = 45, nC = 35, • ES 8: nT = 45, nC = 40, p <.05 • ES 9: nT = 100, nC = 120, F = 8.73 • ES 10: nT = 200, nC = 160, t = 5.66 (see electronic document: “Correct ds for 5 effect sizes.doc”) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 28
The effect sizes are weighted by the inverse of the variance to give more weight to effects based on larger sample sizes Variance for mean level comparison is calculated as The standard error of each effect size is given by the square root of the sampling variance SE = vi Weighting for mean-level differences ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 38 38
Enter_w.xls ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 39 39
SE for gain scores Inverse variance for gain scores Weighting for gain scores T1 and T2 scores are dependent so we need to get correlation between T1 and T2 into equation (not always reported) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 40 40
Enter_w.xls XLS ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 41 41
Weight the ES by the inverse of the s.e. The average ES Standard error of the ES Compute the weighted mean ES and s.e. of the ES ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 44 44
Enter_w.xls ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 45 45
Population and sample Population N = ‘size’ m = ‘mean’ d = ‘effect size’ Sample n = ‘size’ m = ‘mean’ d = ‘effect size’ Interval estimates • The “likely” population parameter is the sample parameter ± uncertainty • Standard errors (s.e.) • Confidence intervals (C.I.) ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 49 49
Calculating rs Means and standard deviations (d) Almost all test statistics can be transformed into an standardized effect size “r” c2f r P-values F-statistics t-statistics “other” test statistics ESRC RDI One Day Meta-analysis workshop (Marsh, O’Mara, Malmberg) 50