The logic and practice of research synthesis 8 th july 2005
1 / 57

- PowerPoint PPT Presentation

  • Uploaded on

Methods for Research Synthesis. The logic and practice of research synthesis' — 8 th July 2005 —. David Gough EPPI-Centre, Social Science Research Unit, Institute of Education, University of London. Today’s seminar. Welcome and introduction Purpose, principles and potential

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '' - millie

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The logic and practice of research synthesis 8 th july 2005 l.jpg

Methods for Research Synthesis

The logic and practice of research synthesis' — 8th July 2005 —

David Gough

EPPI-Centre, Social Science Research Unit, Institute of Education, University of London

Today s seminar l.jpg
Today’s seminar

  • Welcome and introduction

  • Purpose, principles and potential

  • Small group work: exploring selected review tools and procedures

  • Ways forward

Aims of this seminar l.jpg
Aims of this seminar

  • have a critical understanding of the value of systematic methods for synthesizing research evidence;

  • have an understanding of the importance of reviews being question-led;

  • have discussed the diversity of approaches to synthesis and identified principles and decision points central to all;

  • considered the potential for systematic approaches to your own review work.

Social science research unit l.jpg
Social Science Research Unit

Five streams of work:

  • Childhood Studies

  • Evaluation of Social Interventions

  • Sexual Health, Reproduction and Social Exclusion

  • Evidence for Policy and Practice Information and Co-ordinating Centre

  • Perspectives, Participation and Research

Eppi centre personnel l.jpg
EPPI-Centre personnel

David Gough (Director), Ann Oakley (Founding Director)

Angela Harden and Sandy Oliver (Co-directors HP & PH)

Jo Garcia (User involvement, Education)

Jon Ashton

Mukdarut Bangpan

Jackie Barry

Ginny Brunton

Jeff Brunton

Helen Burchett

Esther Coren

Kelly Dickson

Adam Fletcher

Zoe Garrett

Nicholas Houghton

Josephine Kavanagh

Eva Moran

Mark Newman

Chloe Powell

Rebecca Rees

Abigail Rowe

Katy Sutcliffe

James Thomas

Carole Torgerson

Alex Trouton

Helen Tucker

Lisa Underwood

Eppi centre vision and mission statement l.jpg
EPPI-Centre vision and mission statement

Developing and promoting participatory and user-friendly systematic reviews addressing important questions in different domains of policy, practice and research in the public interest

Eppi centre external review groups rgs and teams l.jpg
EPPI-Centre external Review Groups (RGs) and teams

  • 16 funded by the Department for Education and Skills (DfES) on different aspects of teaching and learning (some with joint funding – e.g. Nuffield Foundation, NUT (HEFCE), ESRC TLRP)

  • 8 funded by the Teacher Training Agency on different aspects of teacher education

  • 1 on transitions of students with disabilities from school to work funded by (based in Colorado in the US)

  • 1 in higher education funded by the Learning Skills Development Agency

  • New collaborations with SCIE and DWP

Eppi centre in house reviews l.jpg
EPPI-Centre ‘in-house’ reviews

  • The effect of travel modes on children’s mental health, cognitive, social development (DETR)

  • The effects of personal development profiling for improving student learning (LTSN)

  • Support for pupils with emotional and behavioural difficulties (NFER)

  • Effect of secondary school size on outcomes (DfES)

  • Series of reviews on health promotion and public health (HP & PH) topics funded primarily by the Department of Health (DH) (see next slide)

Health promotion public health review series l.jpg
Health promotion & public health review series


Workplace health promotion

Peer-delivered health promotion for young people


Young people and smoking

Older people and accidents

Young people and sexual health

Men who have sex with men (MSM) and sexual health


Young people and:

…mental health

…physical activity

…healthy eating


Incentives for young people

Active transport

Teenage pregnancy and parenting

Risk behaviour and accidental injury


Children and:

…physical activity

…healthy eating

HIV-health promotion for MSM

Purpose principles and potential l.jpg
Purpose, principles and potential

  • Why use systematic approaches to reviewing?

  • What are the key principles of systematic research synthesis (SRS)?

Imaginary scenario l.jpg
Imaginary scenario

  • You’re in a PTA AGM. The head teacher is speaking about homework. In the room are teachers, parents (one of whom is also a researcher) and school governors.

  • Someone says “I’ve read a literature review that says homework is bad for children”.

  • What might it be useful to know about this review?

  • Nature of variation between summaries of research

Key features of srs l.jpg
Key features of SRS

  • Explicit research question

  • Explicit and transparent method: protocol up front

  • A piece of research: methods to avoid bias

  • Explicit reporting: accountable, replicable, updateable

  • Judgement of study quality and relevance: for question

  • Relevant: need for involvement of research users

Variation between reviews l.jpg
Variation between reviews

6 reviews of older people and accident prevention

Total studies included 137

Common to at least two reviews 33

Common to all six reviews 2

Treated consistently in all reviews 1

From Oliver S, Peersman G, Harden A and Oakley A (1999) Discrepancies in findings from effectiveness reviews: the case of health promotion for older people in accident and injury prevention. Health Ed J, 58:66-77.

Slide14 l.jpg

How some findings evade us

  • Profusion of published and unpublished material

  • Much hidden: only 50% abstracts presented at conferences are later published in full

  • Constraints of own expertise and disciplinary affiliations

Slide15 l.jpg

Some evasion is systematic

  • Some types of studies are harder to find than others*

    • E.g. statistically significant, ‘positive’ results more likely to be:

      • Published, published rapidly

      • Published in English

      • Published more than once

      • Cited by others

  • *work cited in Egger M et al (2003) . How important are comprehensive literature searches and the assessment of trial quality in systematic reviews? Empirical study. Health Technology Assessment 2003; Vol. 7(1)

Slide16 l.jpg

Are systematic searches important?

Some, limited evidence from health about how this affects estimates of effect:

  • Unpublished trials show less beneficial effects than published trials

  • Non-English language trials and non-Medline indexed trials tend to show larger treatment effects

  • Trials that are difficult to locate tend to be smaller and of lower methodological quality

*Egger et al (2003) op cit.

Different searches find different studies 1 l.jpg
Different searches find different studies (1)

How are decisions made about the entry of people aged 65+ to care services?

Taylor B, Dempster M and Donnelly M (2003) Hidden Gems: systematically Searching Electronic Databases for Research Publications for Social Work and Social Care. C J Social Work, 33:423-429.

Different searches find different studies 2 l.jpg
Different searches find different studies (2)

*Adapted from: Harden A, Peersman G, Oliver S, Oakley A (1999) Identifying primary research on electronic databases to inform decision-making in health promotion: the case of sexual health promotion. Health Education Journal 58: 290–301.

Slide19 l.jpg

A matter of degree? Discuss..

Table 1. Systematic and traditional reviews compared

Pettigrew, M. (2001) Systematic reviews BMJ, 322,98-101

What is a systematic review l.jpg
What is a systematic review?

A systematic review develops an evidence-based statement in a particular area of interest. It is a piece of research in its own right using explicit and transparent methods following a standard set of procedures. This means it can be replicated. It judges the quality and relevance of evidence for a given question.

Research evidence for policy and practice l.jpg
Research evidence for policy and practice

  • “…… policy makers and practitioners who intervene in the lives of other people not infrequently do more harm than good”

    Chalmers I (2003) Trying to do more good than harm in policy and practice: the role of rigorous, transparent, up to date, replicable evaluation. Paper commissioned for the Annals of the American Academy of Political and Social Science.

Slide24 l.jpg

‘Reduce the Risk’ Campaign in the early 1990s in the UK

“The risk of cot death is reduced if babies are NOT put on the tummy to sleep. Place your baby on the back to sleep…. …..Healthy babies placed on their backs are not more likely to choke.”

Research for practice l.jpg
Research for practice

“Teaching is not at present a research-based profession”

Hargreaves D (1996) Teaching as a research-based profession: possibilities and prospects. Teacher Training Agency (TTA) Annual Lecture. London: TTA.

Procedural (craft / apprentice) Vs. declarative (for e.g. research based) knowledge

Access to research l.jpg
Access to research

“..(which is) presented in a form or medium which is largely inaccessible to a non-academic audience; and lack(s) interpretation for a policy-making or practitioner audience”.

Hillage J, Pearson R, Anderson A, Tamkin P (1998) Excellence in Research in Schools. London: Department for Education and Employment/Institute of Employment Studies.

Public debate l.jpg
Public debate

“We are, through the media, as ordinary citizens, confronted daily with controversy and debate across a whole spectrum of public policy issues. But typically, we have no access to any form of a systematic ‘evidence base’ - and therefore no means of participating in the debate in a mature and informed manner”.

AFM Smith (1996) Mad cows and ecstasy: chance and choice in an evidence-based society. Journal of the Royal Statistical Society 159(3): 367-383.

Facilitating the use of research for policy and practice l.jpg
Facilitating the use of research for policy and practice

By preparing systematic reviews of the results of relevant, reliable researchAdditional research, IF systematic reviews of existing research show that this is needed

Key stages and variation l.jpg
Key stages and variation


  • What key stages do the different models of systematic research synthesis (SRS) have in common?

  • How do the different models differ?

  • In what ways do review questions influence review methods?

Slide31 l.jpg

Key decision-making stages in SRS

Form review team

Formulate review question and develop protocol

Search for and screen studies (search strategy)

Describe studies (systematic map of research)

Assess study quality (and relevance)

Synthesise findings (answering review question)

Communicate and engage

Reviewing as a team activity l.jpg
Reviewing as a team activity

  • Using more than one reviewer at key decision-making stages

    • Independent work, followed by consensus

    • Quality assurance of samples

  • Research user participation

    • In setting question to increase relevance

    • Could mean service users, practitioners, policy makers…

Formulating the review question l.jpg
Formulating the review question

  • Provides framework for all other stages of review

  • Specify/clarify conceptual framework

    • Type of question ?

    • Study types ?

    • Settings / populations / phenomena ?

  • Issues to consider

    • Why is the review needed ?

    • Who will use the review ?

Developing the review protocol l.jpg
Developing the review protocol

  • Why do we want one?

  • Protocol contains

    • Review question

    • Definitions of key concepts

    • Explicit inclusion criteria

    • Details of review methods for each stage

    • Details of who’s involved/ how to participate

  • Guidance on developing protocols is available

    • EPPI-Centre, Cochrane, Campbell

What is a systematic search l.jpg
What is a systematic search?

  • Thoughtfully planned

  • Carefully executed

  • Accurately recorded

  • Made explicit

  • Open to scrutiny

  • Can be replicated and extended

Systematic and comprehensive searching l.jpg
Systematic and comprehensive searching

  • Based on conceptual framework and resultant inclusion and exclusion criteria

  • Techniques of searching and retrieval aim to maximise the yield of relevant studies while minimizing the number not relevant to the review

  • Searching and screening stages are generally kept separate

  • Both stages are documented

Sensitivity and specificity l.jpg
Sensitivity and specificity

  • Sensitivity

    • Expresses ability to locate all studies of interest

    • Calculated as the number of relevant studies located as a proportion of all that exist

    • Low sensitivity means that many relevant studies missed by search

  • Specificity

    • Expresses the accuracy of the search strategy in identifying studies of interest

    • Calculated as the proportion of the total number of studies identified by search which are deemed ‘relevant’

    • Low specificity means that the search identified many studies that were not relevant to the review.


Screening then describing studies l.jpg
Screening then Describing studies

  • Screen to check that studies found through search strategy do meet the inclusions criteria

  • Describe the studies = systematic map

Systematic maps and systematic synthesis l.jpg
Systematic maps and systematic synthesis

  • Map: What has been done?

    • maps out research activity (for e.g. broader question with multiple designs)

    • provides context for synthesis

    • research designs in primary studies part of that context

    • Allows potential for narrowing question (inclusion criteria)

  • Synthesis: What is known from what has been done?

Quality assessment of studies l.jpg
Quality assessment of studies

  • To make (and record) judgements about study quality

  • To inform the synthesis

    • Only use findings from studies judged to be of high quality to inform recommendations for policy or practice

    • Qualify findings (‘quality framework’)

    • To examine differences in findings according to quality

Methods for synthesis l.jpg
Methods for synthesis

Tend to build on techniques used in primary research

  • Statistical meta-analysis

    • E.g. numerical effect sizes are combined to create a pooled and weighted overall effect size

  • Narrative ‘empirical’

    • Conceptual framework used to construct and organise a narrative to combine the evidence from each study (evidence could be numerical or textual)

  • Conceptual

    • E.g. Qualitative techniques used to organise and combine conceptual or textual data

Dimensions of difference in synthesis models l.jpg
Dimensions of difference in synthesis models

  • Review/research questions (impact, process, need)

  • Research designs considered relevant

  • Types of data - numerical or textual

  • Quality assessment of different designs

  • Breadth of designs and study focus

  • Variation in contribution of each study to the systematic synthesis

Review research questions l.jpg
Review research questions

  • What are the questions that users want answered?

    • What do people want? (Needs assessment)

    • What works? (Impact/effectiveness)

    • Why/how does it it work? (Process/explanation)

    • What is happening? (Implementation levels)

    • Others…?

  • Different research designs best for addressing different questions (fit for purpose)

Organisations supporting systematic reviewing l.jpg
Organisations supporting systematic reviewing

  • The Cochrane Collaboration – healthcare interventions

  • The Campbell Collaboration – interventions in social work, criminal justice and education

  • The EPPI-Centre – full range of empirical research in education (all phases), health promotion, social care and employment

Quality assessment methods l.jpg


Quantitative (Trials)

Provision of pre- and post- data on outcomes

Provision of data on all outcomes measured

Employment of equivalent control/comparison group

Resulted in ‘high’, ‘medium’ and ‘not sound’/ ‘low’ trials


Qualitative (‘Views’)

Quality of reporting (5 items)

Sufficiency of strategies for reliability/validity (4 items)

Extent to which study findings were rooted in children’s own perspectives (3 items)

Quality-assessment methods

Ways forward l.jpg
Ways forward


  • What else do you want to know about the principles and methods discussed today?

  • If planning to use some SRS methods, what issues (drivers, resources, methodology) might be important?

Scoping mapping and reviewing in depth l.jpg
Scoping, mapping and reviewing in-depth

  • Can select from these approaches for a systematic approach to the following kinds of review:

    • What is the size of the literature?

    • What kinds of literature exist?

    • What can this literature tell us reliably?

Examples of other systematic review sources l.jpg
Examples of other systematic review sources

  • The ESRC UK Centre for Evidence Based Policy and Practice Evidence Network


  • The NHS Centre for Reviews and Dissemination, University of York: Database of Abstracts of Effectiveness (DARE)


  • Bibliographic databases (try combining your topic with ‘meta-analysis’ or ‘systematic review’ or ‘research synthesis’)

Eppi centre dissemination l.jpg
EPPI-Centre Dissemination

  • Web site for all EPPI-Centre reviews:

  • Research Evidence in Education Library (REEL)

    • Web-page accessibility

    • Review Group details

    • Protocols

    • Reviews and user perspectives

    • Primary data underlying reviews

      • Bibliographic information

      • Detailed descriptions and quality assessment

Srs not just medicine and effectiveness l.jpg
SRS: not just medicine and effectiveness

  • Campbell R, Pound P, Pope C, Britten N, Pill R, Morgan M, Donovan J (2003) Evaluating meta-ethnography: a synthesis of qualitative research on lay experiences of diabetes and diabetes care. Social Science and Medicine.

  • Gilbert R, Salanti G, Harden M, See S (2005) Infant sleeping position and the sudden infant death syndrome: systematic review of observational studies and historical review of recommendations from 1940 to 2002. Int J Epidemiol. doi:10.1093/ije/dyi088

  • Petrosino A, Turpin-Petrosino, J Buehler J (2005) "Scared Straight" and other juvenile awareness programs for preventing juvenile delinquency. The Cochrane Database of Systematic Reviews 2005 Issue 2

Contact us l.jpg
Contact us

  • David Gough

  • Rebecca Rees

Slide53 l.jpg

Methodological and conceptual challenges l.jpg
Methodological and conceptual challenges

  • Nature of question and data for review and synthesis

  • Exclusion criteria: (i) breadth; (ii) primary research designs

  • Quality assessment of studies

  • Contribution of studies to synthesis

  • Methods of synthesis and interpretation

  • Synthesis of non-empirical and interpretative qualitative research

  • Operationalising user involvement

Technical structural financial challenges l.jpg
Technical/structural/financial challenges

  • Lack of infrastructure to support systematic research synthesis

    • training, funding, sustainability

  • Time and costs of systematic reviews

    • EPPI-Centre DfES education funded at £2.5m for initial 5 year support for EPPI Centre infrastructure and ‘seed corn’ support for 4 new RGs per year

Ideological political challenges l.jpg
Ideological/political challenges

  • Fear of central political control of research rather than democratization of the research process

  • Confusion between experimental methodologies and systematic research synthesis

  • Polarization of ‘quantitative’ and ‘qualitative’ paradigms

  • Shown to be useful processes and products

  • National and international culture change in research, policy, practice and public involvement in research