1 / 33

Conducting Meta-Analyses

Conducting Meta-Analyses. Marsha Sargeant, M.S. D esign A nd S tatistical A nalysis L aboratory University of Maryland, College Park Department of Psychology. Overview of Presentation. What is a meta-analysis and why is it important?

shilah
Download Presentation

Conducting Meta-Analyses

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Conducting Meta-Analyses Marsha Sargeant, M.S. Design And Statistical Analysis Laboratory University of Maryland, College Park Department of Psychology

  2. Overview of Presentation • What is a meta-analysis and why is it important? • Overview of procedures involved in conducting a quantitative meta-analysis • Database structure • Interpretation of effect sizes

  3. Meta-analysis Definition • A statistical analysis of the summary findings of many empirical studies • It’s quantitative! • Distinct from a meta-review

  4. Background • Empirical findings grew exponentially in the middle 50 years of the 20th century • Multiplied beyond our ability to comprehend and integrate it • Hence a growing need to statistically and technically review, rather than through narrative

  5. Background • Review of practices and methods of research reviewers and synthesizers in the social sciences (Jackson, 1978) • Failure to report methods of reviewing

  6. Benefits of Meta-analyses • Increased statistical power • Identification of sources of variability across studies (e.g., inclusion of moderators) • Detection of biases (e.g., Tower of Babel bias) • Detection of deficiencies in design, analysis, or interpretation Ioannidis & Lau, 1999

  7. Limitations of Meta-analyses • Cannot improve the original studies • Method is frequently misapplied • Can never follow the rules of science • Sources of bias are not controlled Ioannidis & Lau, 1999

  8. Rules of the Game • It is quantitative • There is no arbitrary exclusion of data • File drawer effect • Dissertation research is research too! • Unpublished studies • Meta-analysis seeks general conclusions • It is contradictory to think that we can only compare studies that are the same (if they were the same you wouldn’t need to compare them!) Glass, 2000

  9. Methodological Adequacy of Research Base • Findings must be interpreted within the bounds of the methodological quality of the research base synthesized. • Studies often cannot simply be grouped into “good” and “bad” studies. • Some methodological weaknesses may bias the overall findings, others may merely add “noise” to the distribution. From “Practical Meta-analysis” by D.B. Wilson

  10. Confounding of Study Features • Important study features are often confounding, obscuring the interpretive meaning of observed differences • If the confounding is not severe and you have a sufficient number of studies, you can model “out” the influence of method features to clarify substantive differences From “Practical Meta-analysis” by D.B. Wilson

  11. Meta-analysis Overview • Descriptives • Effect sizes (e.g., correlation coefficients) • Distribution and central tendency summarized • Method section • Databases searched • Journals • What attempts were made to not have a biased search? • Criteria for inclusion • No effect studies Rosenthal, 2005

  12. Meta-analysis Overview • Study quality • Use a weighting system • Use raters and non-dichotomous ratings to avoid weighter bias • Optimally raters should be blind to the results of the study • Ratings can be used as an adjustment on effect size or as a moderator to determine whether quality is related to obtained effect size Rosenthal, 2005

  13. Meta-analysis Overview • Consider independence of studies • Treat non-independent studies as a single study with different dependent variables • Recorded variables • Number, Age, Sex, Education, etc • Volunteer status • Laboratory or field study? • Randomized? • Method of data collection (e.g., interview vs questionnaire) • How constructs are operationalized • etc. Rosenthal, 2005

  14. Meta-analysis Overview • Summarize recorded variables • Study characteristics could all be potential moderators of outcome aside from those with particular meaning for the specific area of research • Effect sizes (there are others) • R • Zr (Fisher’s r-Z transformation) • d family • Cohen’s d • Hedge’s g • Glass’s delta Rosenthal, 2005

  15. Examples of Different Types of Effect Sizes • Standardized mean difference • Group contrast research • Treatment groups • Naturally occurring groups • Inherently continuous construct • Odds-ratio • Group contrast research • Treatment groups • Naturally occurring groups • Inherently dichotomous construct • Correlation coefficient • Association between variables research From “Practical Meta-analysis - The Effect Size” by D.B. Wilson

  16. Interpreting Effect Size Results • Cohen’s “Rules-of-Thumb” • standardized mean difference effect size • small = 0.20 • medium = 0.50 • large = 0.80 • correlation coefficient • small = 0.10 • medium = 0.25 • large = 0.40 • odds-ratio • small = 1.50 • medium = 2.50 • large = 4.30 From “Practical Meta-analysis” by D.B. Wilson

  17. Interpreting Effect Size Results • Rules-of-Thumb do not take into account the context of the intervention • a “small” effect may be highly meaningful for an intervention that requires few resources and imposes little on the participants • a small effect may be meaningful if the intervention is delivered to an entire population (prevention programs for school children) • small effects may be more meaningful for serious and fairly intractable problems From “Practical Meta-analysis” by D.B. Wilson

  18. Meta-analysis Overview • Significance levels recorded • Recorded as the one-tailed standard normal deviates associated with p’s • E.g., p’s of .10, .01., .001 would be recorded as Z’s of 1.28, 2.33, and 3.09

  19. Meta-analysis Overview • Report central tendency • Unwieghted mean effect size • Weighted mean effect size (weighting by size of study – can also use quality or other characteristic of interest) • Median • Proportion of studies showing effect sizes in the expected direction • Report number of studies reported on • Optional: total number of participants on which the weighted mean is based • Optional: median number of participants per obtained effect size

  20. Meta-analysis Overview • Report variability • Standard deviation • Max and min effect size found at the 75th and 25th percentile • If normally distributed, the standard deviation is estimated at .75(Q3-Q1)

  21. Database Structure • Database structures • The hierarchical nature of meta-analytic data • The familiar flat data file • The relational data file • Advantages and disadvantages of each • What about the meta-analysis bibliography? From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  22. Database Structure • Meta-analytic data is inherently hierarchical • Any specific analysis can only include one effect size per study (or one effect size per sub-sample within a study) • Analyses almost always are of a subset of coded effect sizes. Data structure needs to allow for the selection and creation of those subsets From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  23. Example of a Flat Data File Multiple ESs handled by having multiple variables, one for each potential ES. Note that there is only one record (row) per study From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  24. Database Structure Advantages and Disadvantages of a Single Flat File Structure • Advantages • All data is stored in a single location • Familiar and easy to work with • No manipulation of data files prior to analysis • Disadvantages • Only a limited number of ESs can be calculated per study • Any adjustments applied to ESs must be done repeatedly • When to use • Interested in a small predetermined set of ESs • Number of coded variables is modest • Comfort level with a multiple data file structure is low From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  25. Example of Relational Data Structure(Multiple Related Flat Files) Database Structure Study Level Data File Effect Size Level Data File Note that a single record in the file above is “related” to five records in the file to the right From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  26. Example of a More Complex MultipleFile Data Structure Study Level Data File Outcome Level Data File Effect Size Level Data File Note that study 100 has 3 records in the outcomes data file and 6 outcomes in the effect size data file, 2 for each outcome measured at different points in time (Months) Database Structure From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  27. Database Structure Advantages & Disadvantages of Multiple Flat Files Data Structure • Advantages • Can “grow” to any number of ESs • Reduces coding task (faster coding) • Simplifies data cleanup • Smaller data files to manipulate • Disadvantages • Complex to implement • Data must be manipulated prior to analysis (creation of “working” analysis files) • Must be able to select a single ES per study for any given analysis • When to use • Large number of ESs per study are possible From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  28. What about Sub-Samples? • So far I have assumed that the only ESs that have been coded were based on the full study sample • What if you are interested in coding ESs separately for different sub-samples, such as, by gender or SES • Just say “no”! • Often not enough of such data for meaningful analysis • Complicates coding and data structure • Well, if you must, plan your data structure carefully • Include a full sample effect size for each dependent measure of interest • Place sub-sample in a separate data file From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  29. Tips on Coding • Paper Coding • include data file variable names on coding form • all data along left or right margin eases data entry • Coding Directly into a Computer Database From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  30. Example Screen from a ComputerizedDatabase for Direct Coding

  31. Coding Directly into a Computer Database • Advantages • Avoids additional step of transferring data from paper to computer • Easy access to data for data cleanup • Data base can perform calculations during coding process (e.g., calculation of effect sizes) • Faster coding • Disadvantages • Can be time consuming to set up • the bigger the meta-analysis the bigger the payoff • Requires a higher level of computer skill From “Practical Meta-analysis – Database Structure” by D.B. Wilson

  32. Final Comments • Meta-analysis • is a replicable and defensible method of synthesizing findings across studies • often points out gaps in the research literature, providing a solid foundation for the next generation of research on that topic • illustrates the importance of replication • facilitates generalization of the knowledge gain through individual evaluations From “Practical Meta-analysis” by D.B. Wilson

  33. Thank You! Email: msargeant@psyc.umd.edu Web: www.umd.academia.edu/MarshaSargeant

More Related