Loading in 2 Seconds...
Loading in 2 Seconds...
Moderator analyses: Categorical models and Meta-regression. Terri Pigott, C2 Methods Editor & co-Chair Professor, Loyola University Chicago firstname.lastname@example.org. Moderator analyses in meta-analysis.
Terri Pigott, C2 Methods Editor & co-Chair
Professor, Loyola University Chicago
A mean effect size and standard error for each group, and then test whether these means are significantly different from one another
The mean effect size and standard error require an estimate of the variance component
QUESTION: Will we assume that each group has the same variance component? Or, will we assume that each group has its own variance component?Categorical moderator models
We believe that the variation among studies is different between groups.
For example, if we are testing out an intervention and we have studies that use either a low-income and a high-income group of students, we might believe that there will be more variation in effectiveness among studies that have mostly low-income participants
Another example: the effectiveness of an intervention for juvenile delinquents will vary more for the group that had a prior arrest than for those that do not have a prior arrestWhat are our assumptions if we decide to use separate estimates within subgroups?
We believe that the variation among effect sizes are the same no matter the group.
For an intervention review, we may assume that the variation among studies does not differ within the groups of interest
Caveat: We might have to use a pooled estimate if we have small sample sizes within subgroups. We need at least 5 cases (in general) to be able to estimate a separate variance component for each subgroupWhat are our assumptions when we use a pooled estimate?
Journals have a significant variance component, and the mean is not different from zero
Dissertations and unpublished studies both have a non-significant variance component, but both find that women score higher on transformational leadership
The test of the variance component as different from zero is exactly the fixed effects test of homogeneity.
To get this test, we compute the test of homogeneity within each group of studies.
Compare 9.09 to a chi-square with df=3-1=2. p-value is 0.011
The assumption made about the random effects variance: separate estimate for each group, or the same estimate for all groups.
Rationale for the choice of variance component
The random effects mean and CI
The value of the variance components (or variance component)
The test of the between-group differences, and its significance
Thus, the precision of each study’s effect size depends on sample size
This is different from our typical application of regression where we assume every person has the same “weight”
Thus, we need to use weighted least squares regression to account for the fact that the precision of each effect size depends on sample sizeRecall that the variance of the effect size
As in a standard regression model, we can use the regression ANOVA table for diagnostics about the fit of a meta-regression
Recall that in a standard regression analysis, we would get the following regression ANOVA table:Test for the fit of the meta-regression model
Qmodelis the test of whether at least one of the regression coefficients (not including the intercept) is different from zero
We compare QMto a chi-square distribution with p – 1 degrees of freedom with p = # of predictors in model
If QM is significant, then at least one of the regression coefficients is different from zeroQM , the model sum of squares
QRis the test of whether there is more residual variation than we would expect IF the model “fits” the data
We compare QRto a chi-square distribution with k - p – 1 degrees of freedom with k = # of studies/effect sizes, and p = # of predictors in model
If QR is significant, then we have more error or residual variation to explain, or that is not accounted for by the variables we have in the modelQR , the error or residual sum of squares
In a standard regression analysis, we find the t-tests on the printout to see which regression coefficients are significantly different from zero
Those significant regression coefficients indicate that these predictors are associated with the outcome
We will use CMA which gives us the z-tests for the regression coefficients
NOTE: When doing meta-regression in a standard program like SPSS, we have to make some adjustments since these programs do not compute the weighted regression in the way we need for meta-analysisTesting significance of individual regression coefficients in meta-regression
0130 Oslo, Norway