1 / 17

Meta-analysis issues

Meta-analysis issues. Carolyn Mair and Martin Shepperd Brunel University, UK. Outline. What is meta-analysis? What is systematic review? Is there a difference? Why the need for meta-analysis MeLLow. Exploit your expertise and knowledge to reach consensus Record responses

zachary
Download Presentation

Meta-analysis issues

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Meta-analysis issues Carolyn Mair and Martin Shepperd Brunel University, UK

  2. Outline • What is meta-analysis? • What is systematic review? • Is there a difference? • Why the need for meta-analysis • MeLLow

  3. Exploit your expertise and knowledge to reach consensus Record responses Discuss further discussed in reverse panel Draft paper disseminated to all for amendment as necessary Details of common understanding of definitions and processes involved will be submitted for publication. What is the difference?

  4. Over to you… • What is a meta-analysis? • What is a systematic review? • What stages are involved? • Is there a difference? • Summarise your responses please

  5. What is meta-analysis? • Combines results of several studies that address a set of related research hypotheses • Attempt to overcome problem of reduced statistical power in studies with small sample sizes (Pearson, 1904) • Educational research meta-analysis (1970s) e.g. Glass, Schmidt, and Hunter • “Meta-analysis is a statistical procedure for synthesising quantitative results from different studies” (Kitchenham, 2004)

  6. What is systematic review? • “Summary of literature that uses explicit methods to perform thorough literature search and critical appraisal of individual studies to identify valid and applicable evidence, and then uses appropriate techniques to combine them” (Wikipaedia, 2006) • A prerequisite for quantitative meta-analysis(Kitchenham, 2004) • Majority based on explicit quantitative meta-analysis of available data, but some are qualitative • Cochrane Collaboration Database of Systematic Reviews • 2,785 complete reviews; 1,625 protocols (September, 2006)

  7. Another definition • 5 major sections of systematic review • Introduction • Searching • Critical Appraisal • Meta-analysis • Discussion • Systematic Review Study Group of Students’ Scientific Research Centre (Tehran University of Medical Sciences)

  8. 7 steps for preparing and maintaining systematic review • Formulating problem • Locating and selecting studies • Critical appraisal of studies • Collecting data • Analyzing and presenting results • Interpreting results • Improving and updating reviews

  9. Why do a review / meta-analysis? • Summarise existing evidence • Identify gaps in current research • Provide framework/background to position new research activities …what differentiates a literature review from a systematic review? …transparency, fairness and completeness

  10. e.g. Basili et al.,1999; Hayes, 1999; Kitchenham et al., 2002, Miller, 1999; Pickard, 1998 Large literature of empirical studies in software engineering cost estimation Have evidence …but it’s not consistent Need to build ‘body of knowledge’ …but heterogeneity of studies, format and provision of data cause problems Meta-analysis improves understanding

  11. …but • Data from individual studies need to be made available in standardised format that enables cross comparison • Need individual data points from all studies

  12. What guidelines are exist? • The Cochrane Reviewer’s Handbook • Guidelines prepared by the Australian National Health and Medical Research Council • ANYTHING ELSE???????????

  13. The MeLLow project • Aim • To develop tool to enable practitioners to select, a priori, the optimal prediction technique for particular circumstances • Objectives • To identify empirical studies of software engineering prediction techniques • Categorise salient project features that impinge on effort • Identify biases and effects within the studies • Problems • Application of range of research methodologies • Outcome • Efforts turned to promoting more rigorous, structured and uniform approach to research in software engineering prediction techniques

  14. Heterogeneity • Accuracy measures • 20 studies (1-16 datasets, 1-6 accuracy measures) (Mair and Shepperd, 2004) • Response variables • Range of reporting techniques and content • Raw data, cleaned data, summary data • Different imputation methods • ignored or imputed (various imputation methods) • Different holdout strategies training set (used to develop prediction system) + validation set (data points to assess prediction system accuracy). e.g. 15%, 22% or 0%

  15. …and • Pet techniques (Spiegelhalter, 1995) • e.g. CBR, GA, RI • Publication bias and the ‘file drawer problem’ (Rosenthal, 1979) ...differences also in understanding what a meta-analysis is!

  16. Advantages of a systematic approach • Provide information about effects across range of settings and methods • Examine evidential support or contradiction of theoretical hypotheses  generate new hypotheses • Consistent study results = robust, transferable phenomenon • Inconsistent study results can be investigated • Furthermore, meta-analysis can try to find effects and biases not evident in individual smaller studies

  17. Terminology and process • What? • How? • Are there common standards?

More Related