1 / 49

Meta-analysis

Session 2.4: 3-level meta-analyses. Funded through the ESRC’s Researcher Development Initiative. Meta-analysis. Department of Education, University of Oxford. Session 2.4: 3-level meta-analyses. Steps in a meta-analysis. Odds Ratio effect sizes . Effect size calculation.

devorit
Download Presentation

Meta-analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Session 2.4: 3-level meta-analyses Funded through the ESRC’s Researcher Development Initiative Meta-analysis Department of Education, University of Oxford

  2. Session 2.4: 3-level meta-analyses Steps in a meta-analysis

  3. Odds Ratio effect sizes

  4. Effect size calculation The odds-ratio is based on a 2 by 2 contingency table The Odds-Ratio is the odds of success in the treatment group relative to the odds of success in the control group (in the present application males and females)

  5. Effect size calculation XLS

  6. Gender bias in peer reviews

  7. Gender Differences in Peer Review** Bornmann, L. (2007). Bias cut. Women, it seems, often get a raw deal in science—So how can discrimination be tackled? Nature, 445(7127), 566.** Bornmann, L., Mutz, R. & Daniel, H. D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1, 226–238. Abstract: Narrative reviews of peer review research have concluded that there is negligible evidence of gender bias in the awarding of grants based on peer review. Here, we report the findings of a meta-analysis of 21 studies providing, to the contrary, evidence of robust gender differences in grant award procedures. Even though the estimates of the gender effect vary substantially from study to study, the model estimation shows that all in all, among grant applicants men have statistically significant greater odds of receiving grants than women by about 7%

  8. Gender Differences in Peer Review Bornmann et al. conducted a multilevel to meta-analysis based peer reviews (grant applications & fellowship applications): 66 effect sizes from 21 studies and a total of 353,725 applications. They found a statistically significant but small effect in favour of men (an effective odds-ratio of 1.07). However, there was systematic variation in the effect sizes beyond random sampling error, suggesting that their results were not generalizable. Typically the next step would be to consider moderators: type of application (grants vs. pre- & post-doctoral fellowships), discipline, or country. However, they noted that : “Unfortunately, the inclusion of one or more of these characteristics into the calculation of the meta-analysis resulted in models that did not converge in the estimation process. This finding indicated that the model estimation became too complex by considering specific interaction effects or the included characteristics had no influence on the outcome, respectively.”

  9. Gender Differences in Peer Review(Meta-analysis data from Bornmann, Mutza, & Daniel,, 2007 Are women disadvantaged in Peer Reviews? Based on 66 outcomes from 21 studies, we evaluate whether there are systematic gender differences in success. Important moderator variables include type of peer review (grants, fellowships), discipline, country, and year. 9

  10. Gender Differences in Peer Review 10

  11. Peer Review Mean: Unwted Study Level

  12. Peer Review Mean: Wted by Ntot

  13. Preliminary Box Plots: Total Sample 90th %tile 10th %tile Median Weighted Unweighted Potential Outliers No Gender Difference 75th %tile Median Effect Size slightly in favour of women Median Effect Size slightly in favour of men 25th %tile Potential Outliers

  14. Preliminary Box Plots: Type No Gender Difference

  15. Preliminary Box Plots: Country No Gender Difference

  16. Preliminary Box Plots: Discipline No Gender Difference

  17. Preliminary Box Plots: Year No Gender Difference

  18. Random effects models

  19. Random effects assumptions • Is only a sample of studies from the entire population of studies to be considered. As a result, do want to generalise to other studies not included in the sample (e.g., future studies). • Variability between effect sizes is due to sampling error plus variability in the population of effects. • In contrast to fixed effects models, there are 2 sources of variance • Effect sizes are independent.

  20. Random effects Where dj is the observed effect size in study j δis the mean ‘true’ population effect size uj is the deviation of the true study effect size from the mean true effect size and ej is the residual due to sampling variance in study j

  21. Conducting random effects meta-analysis • Like the fixed effects model, there are 2 general ways of conducting a random effects meta-analysis: ANOVA & multiple regression • The analogue to the ANOVA homogeneity analysis is appropriate for categorical variables • Looks for systematic differences between groups of responses within a variable • Multiple regression homogeneity analysis is more appropriate for continuous variables and/or when there are multiple variables to be analysed • Tests the ability of groups within each variable to predict the effect size • Can include categorical variables in multiple regression as dummy variables

  22. SPSS Commands

  23. MeanES MACRO Conclusions: Small effect size based on both Fixed & Random models. Slightly in favour of females for Fixed effects, slightly in favour of males for random effects Significant study-to-study variation so random effects and search for moderators appropriate.

  24. Conclusions: Grants: NS effect in favour of women; Fellowships: significant effect in favour of men (but varies from study-to-study); Within variance NS but Var Comp significant; some study-to-study variation remains (particularly in fellowship applications); MetaF MACRO (ANOVA)Type (0=grants, 1=fellowships)

  25. Conclusions: Significant differences in favour of men for biomedical (2) and Social Sciences (3); other disciplines NS MetaF MACRO (ANOVA)Discipline: 1=Phys 2=Biomed 3=SocSc 4=Mult 5=human

  26. MetaF MACRO (ANOVA): Country: 1=Australia 2=Canada 3=Germany 4=Europe 5=Netherlands 6=Sweden 7=UK 8=USA Sweden Conclusions: BIG difference in favour of men in Sweden; smaller differences in favour of men in Germany and Europe; NS differences for other countries.

  27. Multilevel Models

  28. Website Address to get MLWIN Harvey Goldstein developed the MLWIN statistical package used here and has made many contributions to multilevel modeling, including meta-analysis.

  29. 1 Setting Up Meta-analysis 2 • Click on the equation • make logOR the “y” variable • indicate a three level model with L3=study, L2=id, L3=LogOR • Click “done” button 3 4

  30. 1 2 3 Setting Up Meta-analysis • Click “Cons” in the equation • Tick “Fixed Parameter” “(study)” & “i(d)” but not “logOR” • Click the “done” button

  31. 1 2 3 4 Setting Up Meta-analysis • Now click “add term” button • This will bring up the “X-Variable” select SE (the standard error computed earlier) • Tick only the “logOR” box • Click “done”

  32. 1 1 Setting Up Meta-analysis Now we want to constrain the variance at level 1 to be fixed at 1.0. Under “model” select “constrain parameters”; will bring up “parameter constraint” window

  33. 2&3 6 4 5 1 Setting Up Meta-analysis In the parameter constraint window: 1. Click the “random” button; 2.Change “logOR: SE/SE” to 1; 3. Change “to equal” to 1”; 4. “store” the constraints in the first empty column (here “C27”); 5. Click the “attach random constraints” button; 6. Close “Parameter Constraint” Window

  34. Conclusion: The mean effect size (-.101/.040) is significant. The chi-square (389.88) is signif; there is study-to-study variation. explore moderator variables “null” model with no predictors ->pred c50->calc c51=(('logOR'-c50)/'se')**2->sum c51 to b1 = 389.88 ->cprob b1 65 = 5.6052e-045 After Closing the “parameter constraint” window (last slide) Click on “start” button in “equation” window (may have to click estimates button to get values). Compute chi-square value in command interface window

  35. Add “Type” (0=grant, 1=fellow) Conclusion: The effect of type (-.196/.052) is highly significant The mean effect size (-.007/.034) NS for Type = grant (intercept) chi-sq (171.34) signif; remaining study-to-study variation.

  36. Add “DISC”: 1=Phys 2=Biomed 3=SocSc *4=Mult 5=human ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1 = 188.59 ->cprob b1 61 = .1875e-014 Conclusion: The effect of DISC is highly significant (change in chi-sq =389.88 -188.59 = 200.29 (df = 4). Men signif more successful than women in SocSci (relative to multidis, the reference category that is NS.

  37. Add “CNTRY”: 1=Australia 2=Canada 3=Germany 4=Europe 5=Netherlands 6=Sweden 7=UK 8=USA >pred c50->calc c51 = (('logor'-c50)/'se')**2->sum c51 to b1 = 158.76 ->cprob b1 58 = 1.7144e-011 Conclusion: The effect of CNTRY is highly significant (change in chi-sq =389.88 -158.76 = 189.59 (df = 7). Men signif more successful is Swenden (but note large SE) and Germany relative to US (reference category which is NS).

  38. Add “Year” ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1 = 344.02 ->cprob b1 64 = 6.6568e-038 Conclusion: The Linear effect of YEAR is NS. Notice that I changed the intercept to be 2000 (rather than “0” – which is completely out of the range.

  39. Add “Type” & “DISC”: 1=Phys 2=Biomed 3=SocSc *4=Mult 5=human Note that solution is technically improper (study level constrained to be non-negative) ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1 = 105.47 ->cprob b1 60 = 0.00026315 Conclusion: General pattern of results for each variable considered separately still evident. Reference category (Type = grants, Disc = Multi) still NS. Results should be interpreted cautiously because improper solution.

  40. Add “Type” x “DISC” Interact1=Phys 2=Biomed 3=SocSc *4=Mult 5=human Note that solution is technically improper (study level constrained to be non-negative) ->pred c50->calc c51 = (('logor' - c50)/'se')**2->sum c51 to b1= 103.80 ->cprob b1 56 = 0.00010859 Conclusion: The change in chi-sq is NS, suggesting that there is no interaction. Results should be interpreted cautiously because improper solution.

  41. Add “Type” & “CNTRY”: 1=Australia 2=Canada 3=Germany 4=Europe 5=Netherlands 6=Sweden 7=UK 8=USA Conclusion: General pattern of results similar. Men signif more successful is Sweden (but note large SE) and Germany relative to reference category (US Grants).

  42. Type x CNTRY Interaction1=Australia 2=Canada 3=Germany 4=Europe 5=Netherlands 6=Sweden 7=UK 8=USA Conclusion: The change in chi-sq is NS, suggesting that there is no interaction. Results should be interpreted cautiously because improper solution.

  43. Main Effects of Type, Disc & Country Conclusion: When all main effects are included, Type effect nearly unaffected. However, none of the disc effects are significant, although the Sweden and (marginally) Germany are still significant. Results should be interpreted cautiously because improper solution.

  44. Graphs

  45. Graphs: Caterpillar Plots Caterpillar plot based on L1 residuals. Go to the “model” menu and select “residuals” option. This will bring up the “settings” window. Set “SD (comparative)” to 1.96; 3. Set “level” to “1logOR”; 4. click the “Calc” button; 5. click on the “plot” button to bring up the next window. In the “plot” window select “residual +/1 1.96SD x rank. This brings up the original graph. Clicking on the graph bring up a window to modify the graph (a bit)

  46. Conclusion

  47. Summary • The mean effect size was very small, but significantly in favour of men. However, the results did not generalise across studies (there was study-to-study variation). • The effect size was significantly moderated by the type; it was almost exactly 0 for grants and in favour of men for fellowship applications. This difference was not moderated or mediated by other moderators. • There appeared to be some discipline effects (bias in favour of men in social sciences) and country effects (large bias in favour of men for Sweden). However, when all “main” effects included, discipline effects disappeared. • For Grant Proposals there was no evidence of any effect of gender on outcome.

  48. Software • Purpose-built • Comprehensive Meta-analysis (commercial) • Schwarzer (free, http://userpage.fu-berlin.de/~health/meta_e.htm) • Extensions to standard statistics packages • SPSS, Stata and SAS macros, downloadable from http://mason.gmu.edu/~dwilsonb/ma.html • Stata add-ons, downloadable from http://www.stata.com/support/faqs/stat/meta.html • HLM – V-known routine • MLwiN • MPlus

  49. Key references • Bornmann, L. (2007). Bias cut. Women, it seems, often get a raw deal in science—So how can discrimination be tackled? Nature, 445(7127), 566. • Bornmann, L., Mutz, R. & Daniel, H. D. (2007). Gender differences in grant peer review: A meta-analysis. Journal of Informetrics, 1, 226–238. • Cooper, H., & Hedges, L. V. (Eds.) (1994). The handbook of research synthesis (pp. 521–529). New York: Russell Sage Foundation. • Hox, J. (2003). Applied multilevel analysis. Amsterdam: TT Publishers. • Hunter, J. E., & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park: Sage Publications. • Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-analysis. Thousand Oaks, CA: Sage Publications.

More Related