Using meta analyses in your literature review
Download
1 / 58

Using meta-analyses in your literature review - PowerPoint PPT Presentation


  • 302 Views
  • Updated On :

Using meta-analyses in your literature review. BERA Doctoral Workshop 3rd September 2008 Professor Steven Higgins Durham University s.e.higgins@durham.ac.uk. Acknowledgements.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Using meta-analyses in your literature review' - cid


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Using meta analyses in your literature review l.jpg

Using meta-analyses in your literature review

BERA Doctoral Workshop

3rd September 2008

Professor Steven Higgins

Durham University

s.e.higgins@durham.ac.uk


Acknowledgements l.jpg
Acknowledgements

  • This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008-2011.

  • The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas, Institute of Education, London University.

  • The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11).

  • The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

  • You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.


Slide3 l.jpg
Aims

  • To support understanding of meta-analysis of intervention research findings in education;

  • To extend understanding of reviewing quantitative research literature;

  • To describe the techniques and principles of meta-analysis involved to support understanding of its benefits and limitations;

  • To provide references and examples to support further work.


Esrc researcher development initiative l.jpg
ESRC Researcher Development Initiative

  • Quantitative synthesis of intervention research findings in education

    • Collaboration between

      • Durham University

      • York University

      • Institute of Education, London


Why review l.jpg
Why review?

  • Ask the person next to you what the purpose of the literature review is in their thesis

  • See how many different purposes you can think of

  • Join another pair and identify which are the 3 you think are the most important


Why review6 l.jpg
Why review?

  • Summarise existing knowledge

  • What we know, and how we know it

  • For what purpose?

    • Expectation

    • Scenery

    • State of the art (summary)

    • Positioning (conceptual)

    • Progressing knowledge (logic)


The phd literature review l.jpg
The PhD literature review

  • Narrative summary of the area

  • Grand tour of the concepts and terminology

  • Synthesis of empirical findings

  • Background to the study


A systematic review l.jpg
A systematic review

  • is usually more comprehensive;

  • is normally less biased, being the work of more than one reviewer;

  • is transparent and replicable

    (Andrews, 2005)


Examples of systematic reviews l.jpg
Examples of systematic reviews

  • EPPI Centre

    • UK based - wide range of educational topics

  • The Campbell Collaboration

    • 5 education reviews

  • Best Evidence Encyclopedia

    • John’s Hopkins’ - aimed at practice


Systematic reviewing l.jpg
Systematic reviewing

  • Key question

  • Search protocol

  • Inclusion/exclusion criteria

  • Coding and Mapping

  • In-depth review (sub-question)

  • Techniques for systematic synthesis


Systematic reviews l.jpg
Systematic reviews

  • Research and policy

  • Specific reviews to answer particular questions

    • What works? - impact and effectiveness research with a tendency to focus on quantitative and experimental designs


Literature reviewing conceptual relations l.jpg
Literature reviewing - conceptual relations

Narrative review

Systematic review

Meta-analysis


Meta analysis l.jpg
Meta-analysis

  • Synthesis of quantitative data

    • Cumulative

    • Comparative

    • Correlational

  • “Surveys” educational research (Lipsey and Wilson, 2001)


Origins l.jpg
Origins

1952: Hans J. Eysenck concluded that there were no favorable effects of psychotherapy, starting a raging debate which 25 years of evaluation research and hundreds of studies failed to resolve

1978: To proved Eysenck wrong, Gene V. Glass statistically aggregated the findings of 375 psychotherapy outcome studies

Glass (and colleague Smith) concluded that psychotherapy did indeed work - “the typical therapy trial raised the treatment group to a level about two-thirds of a standard deviation on average above untreated controls; the average person received therapy finished the experiment in a position that exceeded the 75th percentile in the control group on whatever outcome measure happened to be taken”(Glass, 2000). Glass called the method “meta-analysis”

( adapted from Lipsey & Wilson, 2001)


Historical background l.jpg
Historical background

  • Underpinning ideas can be identified earlier:

    • K. Pearson (1904)

      Averaged correlations for typhoid mortality after inoculation across 5 samples

    • R. A. Fisher (1944)

      “When a number of quite independent tests of significance have been made … although few or none can be claimed individually as significant, yet the aggregate gives an impression that the probabilities are on the whole lower than would often have been obtained by chance” (p. 99).

      Source of the idea of cumulating probability values

    • W. G. Cochran (1953)

      Discusses a method of averaging means across independent studies

      Set out much of the statistical foundation for meta-analysis (e.g., Inverse variance weighting and homogeneity testing)

      ( adapted from Lipsey & Wilson, 2001)


Significance versus effect size l.jpg
Significance versus effect size

  • Traditional test is of statistical ‘significance’

  • The difference is unlikely to have occurred by chance

    • However it may not be:

      • Large

      • Important, or even

      • Educationally ‘significant’


The rationale for using effect sizes l.jpg
The rationale for using effect sizes

  • Traditional reviews focus on statistical significance testing

    • Highly dependent on sample size

    • Null finding does not carry the same “weight” as a significant finding

  • Meta-analysis focuses on the direction and magnitude of the effects across studies

    • From “Is there a difference?” to “How big is the difference?”

    • Direction and magnitude represented by “effect size”


Effect size l.jpg
Effect size

  • Comparison of impact

  • Same AND different measures

  • Significance vs effect size

    • Does it work? vs How well does it work?


Effect size19 l.jpg
‘Effect size’

  • Standardised way of looking at gain scores

  • Different methods for calculation

  • Experimental group mean - Control mean/ Standard deviation


What is effect size l.jpg
What is “effect size”?

  • Standardised way of looking at difference

    • Different methods for calculation

      • Odds Ratio

      • Correlational (Pearson’s r)

      • Standardised mean difference

        • Difference between control and intervention group as proportion of the dispersion of scores


Calculating effect size l.jpg
Calculating effect size

  • Control group gain minus experimental group gain divided by the standard deviation of the groups


Effect size and impact l.jpg
Effect size and impact

From: Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, Colorado, Mid-continent Regional Educational Laboratory. Available at: http://www.mcrel.org:80/topics/products/83/ (accessed 2/9/08).


Interpreting effect sizes l.jpg
Interpreting effect sizes

  • Relative effects - average is about 0.37 - 0.4(Sipe and Curlette, 1997; Hattie, Biggs and Purdie, 1996)

  • Doing something different makes a difference

  • Visualising the difference


How much is the impact l.jpg
How much is the impact?

0.1 = percentile gain of 6 points

ie a class ranked 50th in a league table of 100 schools would move from 50th to about 44th place

0.5 = percentile gain of 20 points

ie move from 50th to 30th place

1.0 = percentile gain of 34 points

ie move from 50th to 16th place


Other interpretations l.jpg
Other interpretations

0.2 “small” = difference in height between 15-16 year olds

0.5 “medium” = difference in height between 14 and 18 year olds

0.8 “large” = difference in height between 13 and 18 year olds

Cohen 1969


Meta analysis26 l.jpg
Meta-analysis

  • Key question

  • Search protocol

  • Inclusion/exclusion criteria

  • Coding

  • Statistical exploration of findings

    • Mean

    • Distribution

    • Sources of variance


Some findings from meta analysis l.jpg
Some findings from meta-analysis

Pearson et al. 2005

  • 20 research articles, 89 effects ‘related to digital tools and learning environments to enhance literacy acquisition’. Weighted effect size of 0.489 indicating technology can have a positive impact on reading comprehension

    Bernard et al. 2004

  • Distance education and classroom instruction - 232 studies, 688 effects - wide range of effects (‘heterogeneity’); asynchronous DE more effective than synchronous


More findings l.jpg
More findings

Hattie and Timperley, 2007

  • ‘The Power of Feedback’, synthesis of other meta-analyses on feedback to provide a conceptual review 196 studies, 6972 effects - average effect of feedback on learning 0.79


Rank or guess some effect sizes l.jpg
Rank (or guess) some effect sizes…

Formative assessment

CASE (Cognitive Acceleration Through Science Education)

Individualised instruction

ICT

Homework

Direct instruction


Rank order of effect sizes l.jpg
Rank order of effect sizes

1. 04 CASE (Cognitive Acceleration Through Science Education) (Boys science GCSE - Adey & Shayer, 1991)

0.6 Direct instruction (Sipe & Curlette, 1997)

0.43 Homework (Hattie, 1999)

0.32 Formative assessment (KMOFAP)

0.31 ICT (Hattie, 1999)

0.1 Individualised instruction (Hattie, 1999)


Super syntheses l.jpg
‘Super-syntheses’

  • Syntheses of meta-analyses

  • Relative effects of different interventions

  • Assumes variation evens out across studies with a large enough dataset (Marzano/Hattie) or attempts to control for the variation statistically (Sipe & Curlette)


Hattie biggs and purdie 1996 l.jpg
Hattie Biggs and Purdie, 1996

Synthesis of study skills interventions

Meta-analysis of 51 studies of study skills interventions. Categorised the inverventions using the SOLO model (Biggs & Collis, 1982), classified studies into four hierarchical levels of structural complexity and as either ‘near’ or ‘far’ transfer. The results support situated cognition, and that training for other than simple mnemonic tasks should be in context, use tasks within the same domain as the target content, and promote a high degree of learner activity and metacognitive awareness.

(average effect 0.4)


Sipe and curlette 1997 l.jpg
Sipe and Curlette, 1997

  • “A metasynthesis of factors relating to educational achievement” - testing Walberg’s ‘educational productivity’ model - synthesis of 103 meta-analyses


Marzano 1998 l.jpg
Marzano, 1998

‘Theory driven’

Self system - metacognition - cognition/ knowledge

Self - 0.74

Metacogntive 0.72

Cognitive 0.55


Discussion l.jpg
Discussion

  • Work with a colleague to put the statements in order of how comparable you think the research findings are

  • Join another pair (or pairs) and decide how comfortable would you be with comparing the findings


Issues and challenges in meta analysis l.jpg
Issues and challenges in meta-analysis

  • Conceptual

    • Reductionist - the answer is 42

    • Comparability - apples and oranges

    • Atheoretical - ‘flat-earth’

  • Technical

    • Heterogeneity

    • Publication bias

    • Methodological quality


Reductionist or flat earth critique l.jpg
Reductionist or ‘flat earth’ critique

The “flat earth” criticism is based on Lee Cronbach’s assertion that a meta-analysis looks at the “big picture” and provides only a crude average. According to Cronbach,

“… some of our colleagues are beginning to sound like a Flat Earth Society. They tell us that the world is essentially simple: most social phenomena are adequately described by linear relations; one-parameter scaling can discover coherent variables independent of culture and population; and inconsistencies among studies of the same kind will vanish if we but amalgamate a sufficient number of studies…The Flat Earth folk seek to bury any complex hypothesis with an empirical bulldozer…” (Cronbach, 1982, in Glass, 2000).


Comparability l.jpg
Comparability

  • Apples and oranges

    • Same test

    • Different measures of the same construct

    • Different measures of different constructs

    • What question are you trying to answer?

    • How strong is the evidence for this?

“Of course it mixes apples and oranges; in the study of fruit, nothing else is sensible; comparing apples and oranges is the only endeavor worthy of true scientists; comparing apples to apples is trivial” (Glass, 2000).


Empirical not theoretical l.jpg
Empirical not theoretical?

  • What is your starting point?

  • Conceptual/ theoretical critique

    • Marzano

    • Hattie

    • Sipe and Curlette


Technical issues l.jpg
Technical issues

  • Interventions

  • Publication bias

  • Methodological quality

  • Sample size

  • Homogeneity/ heterogeneity


Interventions l.jpg
Interventions

  • “Super-realisation bias” (Cronbach & al. 1980)

    • Small-scale interventions tend to get larger effects

    • Enthusiasm, attention to detail, quality of personal relationships


Publication bias l.jpg
Publication bias

  • Statistically significant (positive) findings

  • Smaller studies need larger effect size to reach significance

  • Larger effects

    • ‘Funnel plot’ sometimes used to explore this

      Scatterplot of the effects from individual studies (horizontal axis) against a study size (vertical axis)


Methodological quality l.jpg
Methodological quality

  • Traditional reviews privilege methodological rigour

    • Low quality studies higher effect sizes (Hattie Biggs & Purdie, 1996)

    • No difference (Marzano, 1998)

    • High quality studies, higher effect sizes (Lipsey & Wilson, 1993)

  • Depends on your definition of quality


Sample size l.jpg
Sample size

“Median effect sizes for studies with sample sizes less than 250 were two to three times as large as those of larger studies.” (Slavin & Smith, 2008)


Heterogeneity l.jpg
Heterogeneity

  • Variation in effect sizes

  • Investigate to find clusters (moderator variables)

  • Assumption that the effect will be consistent


Questions and reactions l.jpg
Questions and reactions

  • With a colleague see if you can identify a question arising from the presentation so far

  • What is your reaction to the technique

  • How useful is it

    • Generally

    • To your own work?


Strengths of meta analysis l.jpg
Strengths of Meta-Analysis

Uses explicit rules to synthesise research findings

Can find relationships across studies which may not emerge in qualitative reviews

Does not (usually) exclude studies for methodological quality to the same degree as traditional methods

Statistical data used to determine whether relationships between constructs need clarifying

Can cope with large numbers of studies which would overwhelm traditional methods of review


Summary l.jpg
Summary

  • “Replicable and defensible” method for synthesizing findings across studies (Lipsey & Wilson, 2001)

  • Identifies gaps in the literature, providing a sound basis for further research

  • Indicates the need for replication in education

  • Facilitates identification of patterns in the accumulating results of individual evaluations

  • Provides a frame for theoretical critique


Other approaches to synthesis l.jpg
Other approaches to synthesis

  • Narrative

  • Quantitative (meta-analysis)

  • Best-evidence synthesis (Slavin)

  • Realist (Pawson)

  • Meta-ethnography (Noblitt & Hare)

  • Thematic synthesis (Thomas & Harden)

  • Grounded theory


Suggestions l.jpg
Suggestions

  • Be explicit about your rationale

  • Be systematic (or at least methodical)

  • Be transparent

  • Describe

  • Analyse (content and methodology)

  • Synthesise


A narrative metaphor l.jpg
A (narrative) metaphor…

  • Literature review as rhetoric

  • An act of persuasion

  • Introduce your study…


Some useful websites l.jpg
Some useful websites

EPPI, Institute of Education, London

http://eppi.ioe.ac.uk/

The Campbell Collaboration

http://www.campbellcollaboration.org/

Best Evidence Encyclopedia, Johns Hopkins

http://www.bestevidence.org/

Best Evidence Synthesis (BES), NZ

http://www.educationcounts.govt.nz/themes/BES

Institute for Effective Education (York)

http://www.york.ac.uk/iee/research/#reviews


Further training l.jpg
Further training

  • ESRC RDI in quantitative synthesis

    • One day training sessions

      • Introduction (for interpretation)

      • Methods Training (for application)

      • Issues Seminars (methodological issues)

    • Durham, London, Edinburgh, Bristol, Belfast, York

      s.e.higgins@durham.ac.uk


References l.jpg
References

Bernard, R.M., Abrami, P.C., Lou, Y., Borokhovski, E., Wade, A., Wozney, L., Wallet, P.A., Fiset, M.,& Huang, B. (2004) How Does Distance Education Compare with Classroom Instruction? A Meta-Analysis of the Empirical Literature Review of Educational Research, 74. 3, (Autumn, 2004), pp. 379-439.

Chambers, E.A. (2004). An introduction to meta-analysis with articles from the Journal of Educational Research (1992-2002). Journal of Educational Research, 98, pp 35-44.

Cronbach, L. J., Ambron, S. R., Dornbusch, S. M., Hess, R.O., Hornik, R. C., Phillips, D. C., Walker, D. F., & Weiner, S. S. (1980). Toward reform of program evaluation: Aims, methods, and institutional arrangements. San Francisco, Ca.: Jossey-Bass.

Glass, G.V. (2000). Meta-analysis at 25. Available at: http://glass.ed.asu.edu/gene/papers/meta25.html (accessed 9/9/08)

Hattie, J. A. (1992). Measuring the effects of schooling. Journal of Education, 36, pp 5-13

Hattie, J., Biggs, J. and Purdie, N. (1996) Effects of Learning Skills Interventions on Student Learning: A Meta-analysis Review of Educational Research 66.2 pp 99-136.

Hattie, J.A. (1987) Identifying the salient facets of a model of student learning: a synthesis of meta-analyses International Journal of Educational Research, 11 pp 187- 212.

Hattie, J. & Timperley, H. (2007) The Power of Feedback Review of Educational Research 77. 1, pp. 81–112.

Lipsey, Mark W., and Wilson, David B. (2001). Practical Meta-Analysis. Applied Social Research Methods Series (Vol. 49). Thousand Oaks, CA: SAGE Publications.

Marzano, R. J. (1998) A Theory-Based Meta-Analysis of Research on Instruction. Aurora, Colorado, Mid-continent Regional Educational Laboratory. Available at: http://www.mcrel.org:80/topics/products/83/ (accessed 2/9/08).

Pearson, D.P., Ferdig, R.E., Blomeyer, R.L. & Moran, J. (2005) The Effects of Technology on Reading Performance in the Middle-School Grades: A Meta-Analysis With Recommendations for Policy Naperville, Il: University of Illinois/North Central Regional Educational Laboratory .

Sipe, T. & Curlette, W.L. (1997) A Meta-Synthesis Of Factors Related To Educational Achievement: A Methodological Approach To Summarizing And Synthesizing Meta-Analyses International Journal of Educational Research 25. 7. pp. 583-698.

Slavin, R.E. and Smith, D. (2008) Effects of Sample Size on Effect Size in Systematic Reviews in Education Paper presented at the annual meetings of the Society for Research on Effective Education, Crystal City, Virginia, March 3-4, 2008.


Acknowledgements58 l.jpg
Acknowledgements

  • This presentation is an outcome of the work of the ESRC-funded Researcher Development Initiative: “Training in the Quantitative synthesis of Intervention Research Findings in Education and Social Sciences” which ran from 2008-2011.

  • The training was designed by Steve Higgins and Rob Coe (Durham University), Carole Torgerson (Birmingham University) and Mark Newman and James Thomas, Institute of Education, London University.

  • The team acknowledges the support of Mark Lipsey, David Wilson and Herb Marsh in preparation of some of the materials, particularly Lipsey and Wilson’s (2001) “Practical Meta-analysis” and David Wilson’s slides at: http://mason.gmu.edu/~dwilsonb/ma.html (accessed 9/3/11).

  • The materials are offered to the wider academic and educational community community under a Creative Commons licence: Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License

  • You should only use the materials for educational, not-for-profit use and you should acknowledge the source in any use.