1 / 55

Meta-analysis

Session 3.3 & 3.4: Teacher Expectancy Example. Funded through the ESRC’s Researcher Development Initiative. Meta-analysis. Department of Education, University of Oxford. Session 3.3 & 3.4: Teacher Expectancy Example. Steps in a meta-analysis. Teacher Expectancy Effects on IQ .

rowland
Download Presentation

Meta-analysis

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Session 3.3 & 3.4: Teacher Expectancy Example Funded through the ESRC’s Researcher Development Initiative Meta-analysis Department of Education, University of Oxford

  2. Session 3.3 & 3.4: Teacher Expectancy Example Steps in a meta-analysis

  3. Teacher Expectancy Effects on IQ (Meta-analysis data from Raudenbush & Bryk, 2002)

  4. Research question • Do teacher expectations influence student IQs? • Teachers led to have high expectations of experimental (through bogus feedback) but not control students. • The focus is on the effect of how long teachers knew students prior to the experimental intervention.

  5. Teacher Expectancy Effects on IQ (Meta-analysis data from Radudenbush & Bryk, 2002 Do teacher expectations influence student IQs? Teachers led to have high expectations of experimental (through bogus feedback) but not control students. Focus here is on the effect of how long teachers knew students prior to the experimental intervention.

  6. Teacher Expectancy Effects on IQ (Meta-analysis data from Radudenbush & Bryk, 2002 6

  7. 75th %tile 90th %tile 10th %tile Median Preliminary box plots Potential Outliers 25th %tile

  8. Fixed and random effects Mean effect size and homogeneity analyses

  9. SPSS Commands

  10. MeanES MACRO • 2523 0 *-------------------------------------------------------------- • 2524 0 *' Macro for SPSS/Win Version 6.1 or Higher • 2525 0 *' Written by David B. Wilson (dwilson@crim.umd.edu) • 2526 0 *' Meta-Analyzes Any Type of Effect Size • 2527 0 *' To use, initialize macro with the include statement: • 2529 0 *' INCLUDE "[drive][path]MeanES.SPS" . • 2530 0 *' Syntax for macro: • 2532 0 *' MeanES ES=varname /W=varname /PRINT=option . • 2534 0 *' E.g., MeanES ES = D /W = IVWEIGHT . • 2535 0 *' In this example, D is the name of the effect size variable • 2536 0 *' and IVWEIGHT is the name of the inverse variance weight • 2537 0 *' variable. Replace D and INVWEIGHT with the appropriate • 2539 0 *' variable names for your data set. • 2540 0 *' /PRINT has the options "EXP" and "IVZR". The former • 2541 0 *' prints the exponent of the results (odds-ratios) and • 2542 0 *' the latter prints the inverse Zr transform of the • 2543 0 *' results. If the /PRINT statement is ommitted, the • 2545 0 *' results are printed in their raw form.

  11. MeanES MACRO

  12. MeanES MACRO Conclusions: Small (NS) effect size based on both Fixed & Random models. Significant unexplained variance suggesting non-generalisability of effects across studies and the need for a random effects model.

  13. Fixed and random effects Analogue to the ANOVA analyses

  14. MetaF MACRO (ANOVA)based on categorical weeks Conclusions: Large effect of weeks (20.4/35.8= 57% var expl); Total Residual, variance component & residual by group all NS; ES significant for 1st two groups, NS last two groups

  15. Fixed and random effects Regression analyses

  16. MetaReg MACRO (Regression)based of categorical weeks Conclusions: Large effect of weeks (54% var expl); Constant term highly significant (at intercept = 0); Residual variance NS (variance component = 0)

  17. MetaReg MACRO (Regression)based on uncategorical weeks Conclusions: Large effect of weeks (21% var expl); Constant term highly significant (at intercept = 0); Residual var NS; Does not do as well as categorised weeks

  18. Fixed and random effects Variations of the previous analyses

  19. MetaF MACRO (ANOVA)based of Blind vs. Aware Test Administrators Conclusions: Small, NS effect; Resid var marginally significant for “Aware” not “blind”

  20. MetaF MACRO (ANOVA)based of Group vs. Individual IQ Tests Conclusions: Small, marginally significant effect (4.1/34.3=12% var expl); ES NS for “Group” but signi for “individual” Resid var for group signif but individual NS; All effects very small

  21. MetaReg MACRO (Regression)based of Group vs. Individual IQ Tests Conclusions: Small, marginally significant effect (12% var expl); Constant term (Group) NS; Resid var signif (but variance component NS)

  22. MetaReg MACRO (Regression)based of (Group vs. Individual IQ Tests) & (categorised weeks) Conclusions: Effect of test type no longer signif when weeks included. Effect of weeks nearly unaffected. Note can only look at multiple variables with regression.

  23. Multilevel models

  24. Website Address to get MLwiN Harvey Goldstein developed the MLwiN statistical package used here and has made many contributions to multilevel modeling, including meta-analysis.

  25. Always a bit dangerous to say some one person invented a new approach. However, fair to say that Stephen Raudenbush at least popularised the multilevel approach to meta-analysis with the meta-analysis of the teacher-expectancy data considered here. Raudenbush, S.W. and Bryk, A.S. (2002).Hierarchical Linear Models (Second Edition).Thousand Oaks: Sage Publications, 482 pp. Raudenbush, S.W. (1984). Magnitude of teacher expectancy effects on Pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments.Journal of Educational Psychology, 76, 1, 85-97.

  26. Getting Data Into MLwiN In an empty MLwiN file, puts the xx input variables into first xx columns. (can also add new data to existing files). Check to see that data is correct and click on “paste” button

  27. Getting Data Into MLwiN Check the MLwiN “names” file to see that data looks ok (e.g., missing values; min & max values).

  28. Setting Up Meta-analysis MLwiN will open an empty equation that you have to construct. Click on the “y” to bring up this screen. select “d” (the effect size) as the dependent variable Select “2” for “N of levels” select “ID” for Level 1 select “d” for Level 2

  29. 1 Setting Up Meta-analysis 3 2 • Click “Add Term” Button (bottom equations window) • Select “cons” (variable = 1 for all cases) • Click the “done” button

  30. 1 2 3 Setting Up Meta-analysis • Click “Cons” in the equation • Tick “Fixed Parameter” & “j(id)” but not “i(d)” • Click the “done” button

  31. 1 2 3 4 Setting Up Meta-analysis • Now click “add term” button • This will bring up the “X-Variable” select SE (the standard error computed earlier) • Tick only the “i(d)” box • Click “done”

  32. 1 1 Setting Up Meta-analysis Now we want to constrain the variance at level 1 to be fixed at 1.0. Under “model” select “constrain parameters”; will bring up “parameter constraint” window

  33. 2&3 1 Setting Up Meta-analysis • In the parameter constraint window: • Click the “random” button • Change “d: SE/SE” to 1 • Change “to equal” to 1”

  34. 1 2 3 Setting Up Meta-analysis • “store” the constraints in the first empty column (“C19”) • Click the “attach random constraints” button. • Close the “Parameter Constraint” Window

  35. “null” model with no predictors Conclusion: The mean effect size (.078) is not significant. The chi-square is significant; there is study-to-study variation. Reasonable to explore moderator variables After Closing the “parameter constraint” window (last slide) Click on “start” button in “equation” window (may have to click estimates button to get values). Compute chi-square value in command interface window

  36. Add raw “weeks” variable Conclusion: The effect of weeks (-.013/.005) is significant The mean effect size (.162/.055) signif (when weeks = 0). chi-sq signif; some remaining study-to-study variation.

  37. WKCAT: 4-category weeks Conclusion: categorized weeks does best of of (chi-sq = 16.568)

  38. Aware vs. Blind Administration For a categorical variable, you choose a reference (“left out” category. Default is the 1st category >pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 35.608 ->cprob b1 17 = 0.0051714 Conclusion: Main Effect of Aware vs. Blind is NS

  39. Aware vs. Blind Administration For a categorical variable, you choose a reference (“left out” category. Default is the 1st category >pred c50; ->calc c51 = (('d' - c50)/'se')**2; ->sum c51 b1 = 16.445 ->cprob b1 17 = 0.49254 Conclusion: Effect for Blind vs. Aware reduced by controlling for “wkcat” but was already nonsignificant

  40. Individual vs. Group Tests Conclusion: Effect seems larger for individually administered tests, but not after control for weeks (wkcat)

  41. Individual vs. Group Tests Order = 1 to specify a 2-way interaction term Variables in interaction Conclusion: No interaction effect (chi-sq little different than wkcat alone (15.114 vs. 16.568). Notice that the effect of test type (and its SE) are very large (.2901/.4859=.597)

  42. Individual vs. Group Tests Grand Mean Centered Grand Mean Centered Conclusion: Same results but estimated & SE for individual term is smaller (reduced multicollinearity by grand mean centering the wkcat variable). Note that chi-sq is the same.

  43. Advanced multilevel analyses

  44. “weeks” centered at 2 Weeks is centered at 2. >pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 28.937 ; ->cprob b1 17 = 0.035115 Conclusion: The effect of weeks (-.013/.005) & chi-sq (28.937) same as with original weeks. The mean effect size (.136/.049) signif (when weeks = 2).

  45. “weeks” centered at 6 & 7 Weeks centered at 6 Weeks centered at 7 >pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 28.937 ; ->cprob b1 17 = 0.035115 Conclusion: Constant term (intercept weeks = 6) is signif (.083/.042=1.98) Constant term (intercept weeks = 7) is NS (.070/.042=1.68)

  46. “weeks” polynomial = 2 >pred c50;->calc c51 = (('d'-c50)/'se')**2; ->sum c51 to b1 = 26.237; ->cprob b1 16 = 0.050779 Conclusion: The linear term is significant but the quad term is not.

  47. “weeks” polynomial = 3 Conclusion: All three polynomial terms are significant and the residual variance component is substantially reduced.

  48. “Log-e weeks+1” Conclusion: The linear term based on the log transform explains more variance than the original (untransformed) weeks (chi-sq = 24.636 vs. 28.937).

  49. WKCAT: Centered at 2 & 3+ Conclusion: Intercept at wkcat = 2 is significant, but intercept at wkcat = 3 is not

  50. 1 Graphs: Caterpillar Plots Caterpillar plot based on L1 residuals. Go to the “model” menu and select “residuals” option. This will bring up the “settings” window. Set “SD (comparative)” to 1.96; 3. Set “level” to “1d”; 4. click the “Calc” button; 5. click on the “plot” button to bring up the next window. In the “plot” window select “residual +/- 1.96SD x rank. This brings up the original graph. Clicking on the graph bring up a window to modify the graph (a bit)

More Related