developing a review protocol searching and data extraction n.
Skip this Video
Download Presentation
Developing a Review Protocol, Searching and Data Extraction

Loading in 2 Seconds...

play fullscreen
1 / 57

Developing a Review Protocol, Searching and Data Extraction - PowerPoint PPT Presentation

  • Uploaded on

Developing a Review Protocol, Searching and Data Extraction. Trudy Bekkering, PhD Center of Evidence-Based Medicine & Belgian Branch of the Dutch Cochrane Center Centre for Methodology of Educational Research Katholieke Universiteit Leuven, Belgium

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Developing a Review Protocol, Searching and Data Extraction' - abra

Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
developing a review protocol searching and data extraction

Developing a Review Protocol, Searching and Data Extraction

Trudy Bekkering, PhD

Center of Evidence-Based Medicine &

Belgian Branch of the Dutch Cochrane Center

Centre for Methodology of Educational Research

KatholiekeUniversiteit Leuven, Belgium

This presentation is partly based on previous presentations of Oliver Wendtand Chad Nye

phases campbell r eview process
Phases Campbell Review Process
  • Register title
  • Develop protocol (project plan)
  • Undertake systematic review
  • Publish systematic review

the review protocol
The Review Protocol
  • A priori statement of aims and methods of the review
  • Research question(s), aims, methods are considered in advance of identifying the relevant literature
    • Conduct review with minimal bias
    • Access to peer review
    • Greater efficiency in review process

(Torgerson, 2003)

r eview protocol advantages
Review Protocol Advantages
  • Clear research question before the review
  • Avoid retrieving irrelevant papers
  • A priori inclusion and exclusion criteria  avoid changing criteria as review progresses or studies may be included on basis of their results
  • If decisions are explicit it enables them to be justified
  • Develop protocol as independently as possible from the literature  avoid influence by one or two major studies, less bias (Torgerson, 2003)
review protocol construction
Review Protocol Construction
  • “The review protocol should ideally be conceived by a group of reviewers with expertise both in the area and the science of research synthesis” (Egger & Davey-Smith, 2001)
  • Prior knowledge of an area can aid the review process, but can also introduce bias to the review  can be reduced by clear, consistent and logical justifications for inclusion criteria
contents protocol
Contents Protocol
  • Title and authors
  • Background
  • Objectives
  • Methods
    • Searching for studies
    • Selecting studies
    • Data extraction (including assessment study quality)
    • Data analysis
  • Acknowledgements / conflicts of interests
  • References
  • Tables / Figures
  • Describes the context of the review: why is it important?
    • Description of the problem
    • Description of intervention
    • How the intervention might work
    • Description potential subgroups
  • Clarify conceptual issues central to the review
objective s
  • Precise statement of objective, using PICO(S)
    • Which population?
    • Which intervention(s)?
    • Which comparison(s)?
    • Which outcome(s)?
    • (Which study design?)

A well formulated objective focuses the review

example objective
Example objective
  • To assess the effects of
  • home-based programmes aimed at
  • improving developmental outcomes for
  • pre-school childrenfromdisadvantagedfamilies

(Miller & Eakin, 2011)

  • How to select studies?
  • How to search for studies?
  • Assessment quality of included studies
  • How to extract data?
  • How to analyse results?
selecting studies
Selecting studies
  • Define inclusion and exclusion criteria
  • Key features:
    • Established a priori
    • Explicit (explain all terms)
    • Based on PICO
    • Include study design, if applicable
    • All retrieved studies are listed in the review either under included or excluded together with a justification
example inclusion criteria
Example inclusion criteria

Types of studies

  • Randomised controlled trials. The control group will either receive no intervention or standard care. Studies comparing two different types of home based programme without a control group will be excluded.
  • This is an area where RCTs are technically and ethically possible. As such this review is confined to RCTs as they provide the best evidence of effectiveness.

(Miller & Eakin, 2011)

example cont
Example (cont)

Types of participants

  • Home based child development interventions for pre-school children from socially disadvantaged families
  • Parents with children up to the age of school entry and who are socially disadvantaged in respect of poverty, lone parenthood or ethnic minority status. Age of school entry can vary between countries (4 to 7 years) and so the upper age range for this review will be the school entry age for the country in which the trial took place.
  • Studies will be excluded if they aim to recruit particular clinical subgroups of parents.
example cont1
Example (cont)

Types of interventions

  • Home based interventions designed to improve child intellectual and socio-emotional development through the provision of relevant knowledge and skills to the parent.
  • The intervention is delivered by trained lay or professional family visitors.
example cont2
Example (cont)

Types of outcomemeasures


The following child outcomes will be included:

  • Cognitive development (including language development and attention)
  • Socio-emotional development (including self regulation and behavioural development)


  • Parentsfeeldisempowered
  • How to select studies?
  • How to search for studies?
  • Assessment quality of included studies
  • How to extract data?
  • How to analyse results?
searching for studies
Searching for studies
  • Preliminary Searches
    • Supports beginning steps: Definition of key concepts & research question
    • Use of standard reference tools and broad searches for review articles and key primary studies
  • Main Searches
    • Identification of primary studies through searches of databases, Web, branching, manual searches
  • Final Searches
    • Refine search terms and update original searches
examples of databases
Examples of Databases
  • Education: ERIC, British Education Index, Australian Education Index, CBCA Education, Education: A SAGE Full-text Collection; Education Full text, Linguistics and Language Behavior Abstracts.
  • Psychology: PsycINFO, PubMed (Medline), Ageline, Psychology: A SAGE Full-Text Collection, Criminology: A SAGE Full-Text Collection
  • Sociology: Sociological Abstracts, Contemporary Women’s Issues. Sociology: A SAGE Full-text Collection
  • Multidisciplinary: Academic Search Premier, ProQuest Dissertations and Theses, FRANCIS, Social Sciences Index, SCOPUS, Web of Science
main searches decisions
Main Searches: Decisions
  • Preparation of Search Strategies
    • What are the key concepts to be searched?
    • How are these represented in each discipline?
    • What are their related terms?
    • How are these key concepts represented in the controlled vocabulary within each database to be searched?
sample research question
Sample Research Question

Approaches to Parent Involvement


Improving the Academic Performance


Elementary School Age Children

Chad Nye, Herb Turner, Jamie Schwartz

The purpose of this review is to determine the effectiveness of parental involvement in improving the academic performance of school age children in grades K-6.

using a thesaurus
Using a Thesaurus
  • From the research question, determine the main concepts to be searched (usually there are three):
    • Intervention: Parental involvement
    • Outcome:Academic performance
    • Population: Kindergarten or Elementary students
  • Consult the main database to be searched.
  • Look up each concept in the thesaurus for this database.
    • A thesaurus is an alphabetical listing of the controlled vocabulary (or descriptors) used within a subject database
    • A hierarchical arrangement is used so that Broader, Narrower and Related headings may be discovered
    • The user will be sent from invalid headings to valid headings.
example eric
Example: ERIC
  • Selecting the ERIC Descriptors
    • Descriptors: Parental Involvement See: Parent participation
    • Related descriptors: Family involvement, Parent-school relationship, Parent role, Parents as teachers
    • Related keywords: parent* involvement, parent* effectiveness, parent* support, family support
    • Descriptors: Academic PerformanceSee: Academic achievement
    • Related descriptors: Science achievement, Reading achievement, Writing achievement, Achievement gains
    • Descriptors: Elementary School Children See: Elementary school students
    • Related descriptors: Elementary education, Primary education, Kindergarten
  • Intervention: Parental involvement
  • Outcome:Academic performance
  • Population: Kindergarten or Elementary students
main searches decisions1
Main Searches: Decisions
  • Construction of the Search Statements
    • What terms should be searched as descriptors or as “keywords”?
    • What Boolean operators should be used?
    • Where should truncation characters be used? (e.g. parent* will retrieve parent, parents, parental)
    • What limiting features are available to narrow results? (e.g. use of Publication Type codes, time period, language)?
boolean operators
Boolean Operators

AND: Both terms must be present in order for a record to be retrieved. Used to combine different concepts.

e.g parent participation AND achievement

OR: Either term may be present in order for a record to be retrieved. Used to search for related terms or synonyms.

e.g. parent OR family

NOT: Used between two terms to ensure that the second term will not appear in any of the results.

e.g. literacy NOT adult

(Parental involvement OR parent participation) AND academic achievement AND (elementary OR primary education)

example eric cont d
Example: ERIC, cont’d
  • Combining Keywords/Descriptors using Boolean operators:
      • DE=(Parent participation OR Family involvement OR Parent role OR Parent-school relationship OR Parents as teachers)
      • “Parent* involvement” OR “Parent* effectiveness” OR “Parent* support” OR “family support”
      • #1 OR #2 = 28,958 records
      • DE=(Academic achievement OR Science achievement OR Reading achievement OR Achievement gains) – 46,574 records
      • DE=(Elementary school students OR elementary education OR elementary schools OR primary education OR kindergarten) - 291997 recs
      • #3 AND #4 AND #5 = 1,669 records
limiting your results
Limiting Your Results

Using the Limiting Commands:

  • Limiting fields contain information that is common to a large number of records within a database. These include language, document type, publication year and so.
  • Some limiting fields will vary across databases (e.g. Classification Code, Age Group)
  • Decisions about whether you are going to restrict:
    • The language of the document
    • The search to a certain time period
    • The results to empirical studies only
example eric cont d1
Example: ERIC, cont’d

Using Limits in ERIC:

    • Search by DT= research -reports
    • Combine this search with the previous one
        • 835/1669recs
        • Review Date Range
        • Review Language

One final comment…

  • Locating ERIC documents:
eric record
ERIC record

AUTHOR : Polovina,-Nada; Stanisic,-Jelena

TITLE: A Study on Family-School Cooperation Based on an Analysis of School Documentation


SOURCE: Online Submission. Journal of Educational Research (Belgrade), v39 n1 p115-133


DOCUMENT TYPE: Journal-Articles; Reports-Research


DESCRIPTORS: Student-Behavior; Attendance-Patterns; Child-Development; Parents-; Family-School-Relationship; Parent-Teacher-Cooperation; Foreign-Countries; Elementary-Schools; Parent-Participation; Grades-Scholastic; Academic-Achievement; Parent-Influence; Parent-Child-Relationship


ABSTRACT: Family-school cooperation is a very complex process that can be studied at different levels in a number of different ways. This study has covered only some aspects of cooperation between parents and teachers, based on school documentation of a Belgrade elementary school. The study covered analyses of 60 Attendance Registers pertaining to 60 classes with 1289 students from Grade 1 through Grade 8 during an academic year. The unit of analysis included: parents attendance at PTA meetings and individual meetings between parents and teachers. In addition to the frequency of parents' visits to school, the relationship between such registered parents' visits and overall academic performance, grades in conduct, excused and unexcused absence from classes were also considered. The research findings indicated interference between development factors (attitude change in parent-child relationship and growing-up) and parents' informal "theory of critical grades" i.e. transitional processes in schooling. The findings confirmed that parents' individual visits to school were mainly meant to offer an excuse for the student's absence from school, while attendance at PTA meetings was linked to poor grades in conduct and missed classes (both excused and unexcused). The findings also showed that parents pursued visiting strategies which were pragmatic, less time-consuming and less emotionally draining ones. The closing part refers to discussions on practical use of the study and possible further research. (Contains 4 graphs.) [This article is the result of the project "Education for Knowledge-Based Society" No. 149001 (2006-2010), financially supported by the Ministry for Science and Environmental Protection of the Republic of Serbia.] (Author)

next steps
Next Steps

Repeat these steps for each database to be searched.

additional retrieval methods
Additional Retrieval Methods
  • The web…to locate grey literature
    • Use Advanced Searchscreens on large engines (eg. Google, Altavista, AlltheWeb, MSN Live)
    • Consult specific sites
  • Branching (Pearl Growing)
    • Scan the bibliographies of previous reviews on your topic
  • Manual searches
    • Browse the Table of Contents of key journals for current years
managing your results
Managing Your Results
  • Export the results
    • Save as a Text file
  • Import into a bibliographic management software:
    • RefWorks,
    • Reference Manager,
    • EndNotes
  • Edit your inhouse database
    • Add Source codefor each database searched (e.g ERIC1, PsycINFO1…)
    • Add notes to the records (e.g.includes vs excludes)
  • Compile a Search Historydocument listing the original search strategies
    • Use of IRMG/ECG Database worksheet
information retrieval wrap up
Information Retrieval: Wrap Up
  • Importance of information retrieval process
    • Not a “one-shot”deal
    • Requires expertise in the planning and implementation of searches
    • Consulting with the Trials Search Advisor or an Information Specialist is highly recommended
  • Use a bibliographic management software
    • Store, manage and organize results
  • Must have ability to replicate review
    • Documentation of entire process, including search strategies used for each database, decisions taken, etc.
  • How to select studies?
  • How to search for studies?
  • Assessment quality of included studies
  • How to extract data?
  • How to analyse results?
data extraction
Data Extraction

What is data extraction?

  • Process of reading through a study and extracting the relevant information from each study
  • The reviewer fills out an data extraction sheet with the appropriate information taken from that study
  • Data extraction sheet may be on paper or electronically
extraction form
Extraction Form
  • Use close-ended items as much as possible (end goal may be to build a database)
  • Design for easy use
    • Cluster items that deal with similar themes and format so that they are easy to complete

(Lipsey & Wilson, 2001)

extraction form cont
Extraction Form (cont.)
  • Extraction form consists of two different parts
    • characteristics that apply to the entire study
    • results: studies typically have multiple outcomes and therefore different effect sizes
extraction manual
Extraction Manual
  • Essential for data extraction process
  • Specifies and describes what study features should be extracted
  • Helps to minimize error and bias in the judgments of the extraction process
  • Facilitate the consistency of extraction
  • Should be pilot-tested with several articles and revised accordingly before actual use
extraction manual cont
Extraction Manual (cont.)
  • Have a full definition of each item and guidelines for extraction
    • Assists the coder in handling ambiguous, unusual cases
  • Bottom line: spell everything out in detail, give full definitions of the various response options, and provide guidelines for dealing with borderline cases
assessment study quality
Assessment study quality
  • Study quality may affect conclusions
  • A systematic review always includes an assessment of individual study quality
  • Different tools available
  • Describe how to use this information
    • Description methodological quality
    • Exclude studies of poor quality?
    • To include in conclusions (strong versus weak evidence)
example study characteristics
Example. Study characteristics
  • Research design
    • Non-randomized
    • Randomized study
      • Pseudo-randomized
      • Cluster-randomized
  • Setting
    • Public school
    • Private school
  • Etc.
example population characteristics
Example. Population characteristics

For each group:

  • Number of participants that were enrolled
  • Number of participants that were randomized
  • Number of participants that completed study
  • Age
  • Gender
  • Ethnicity
  • Baseline reading score
  • Etc.
example intervention characteristics
Example. Intervention characteristics
  • Intervention offered to intervention group
    • Intensity
    • Duration
    • Implementation integrity
  • Intervention offered to control group
    • Intensity
    • Duration
    • Implementation integrity
extraction study results
Extraction study results
  • Descriptive or other statistical information contained in studies needs to be converted into a standard metric by which studies can be compared
  • Continuous outcomes: mean and standard deviation in the groups
  • Dichotomous outcomes: numbers in the groups having the outcome versus the numbers in the groups not having the outcome
extraction form example
Extraction Form Example
  • See example in your package

A. Publication Source

1. Journal 2. Dissertation 3.Thesis

4. Unpublished Paper 5. Book 6. Other: ________

B. Subject Characteristics (page(s): ____________)

Group Pretest Posttest Attrition Age Grade(s) Males

(n) (n) (n) (n) (n) (n)

Tx: _22___ _22__ ______ ______ ______ _____

Cp: ______ _____ ______ ______ ______ _____

Co: _22___ __18_ __ 4___ ______ ______ _____


C. Socioeconomic Status (page(s):___________

  • Low 2. Middle 3. Upper
  • 4. Low-Middle 5. Middle Upper 6. Labeled Mixed
  • Unclear 8. ___________ 9. NR
  • Comments:__________________________________
  • Etc., etc.
bias in data extraction
Bias in data-extraction
  • “Data extraction bias”: reviewer introduces bias into the review
    • Studies in accord with own view
    • Different judgments of quality or methodological adequacy to different studies
    • Reviewer’s awareness of the study authors, the journal, the results  blinding the reviewers
    • Using multiple reviewers
    • Having detailed extraction manual
    • Validating the data extraction
      • Checked by another person
      • Two people working independently (much less common but likely to be more reliable) (Petticrew & Roberts, 2006)
reliability of extraction
Reliability of extraction
  • Reliability has two dimensions:
  • Consistency of a single reviewer
  • Consistency between different reviewers
  • To check  draw subsample of studies, have reviewers extract them again and compare the results
  • For single reviewer reliability: this person needs to extract studies again after sufficient time has passed so that no memory exists of original extraction
  • For between reviewer reliability: different reviewers extract the same sample without reference to what the other has done (Lipsey & Wilson, 2001)
reliability of extraction cont
Reliability of extraction (cont.)
  • Once the appropriate double extraction has been done on a sample, the two sets of results are compared item by item
  • Calculate reliability coefficients:
    • For continuous variables use Pearson’s r (e.g., participant’s age, effect size values)
    • For categorical variables use percentage agreement procedure or Cohen’s kappa (e.g., type of intervention, outcome measure, design)
  • Report actual reliability, then resolve discrepancies

(Lipsey & Wilson, 2001)

training the reviewers
Training the reviewers
  • Effective reviewer must
    • Understand extraction protocol in detail
    • Have the knowledge and skills to properly read and interpret research report
    • Have background in methodology
    • Be familiar with specific research domain
  • Pilot review is recommended
    • Carefully read the manual
    • Extract pilot studies and compare results
    • Discuss disagreements and improve manual
protocol on methods
Protocol on methods

Protocol contains

  • Data extraction form, including manual
  • Method of extraction
    • Paper or electronically (which software)
  • How many reviewers
  • How are reviewers trained
  • How to resolve disagreements
  • Review protocol important first step when undertaking systematic review
  • Helps to focus and structure the review
  • Limits the scope for bias
  • Use standard format – C2 format is recommended


Examples of C2 Review Protocols are available on:

references more reading
References / more reading
  • Egger M & Davey-Smith G. (2001). Principles of and procedures for systematic reviews. In M Egger G Smith & D Altman (Eds.), Systematic Reviews in Health Care: Meta-analysis in Context (2nd ed., pp. 23-42 ). London, UK: BMJ Books.
  • Frost LA & Bondy AS. (2002). The Picture Exchange Communication System. Newark, DE: Pyramid Educational Products.
  • Lipsey MW & Wilson DB. (2001). Practical meta-analysis. Thousand Oaks, CA: SAGE.
  • Miller S & Eakin A. Home based child development interventions for pre-school children from socially disadvantaged families (Protocol). The Campbell Library, 2011.
  • Petticrew M & Roberts H. (2006). Systematic Reviews in the Social Sciences: A Practical Guide. Malden, MA: Blackwell Publishing.
  • Torgerson C. (2003). Systematic Reviews. London, UK: Continuum.
  • Resources Campbell Collaboration:
  • education_articles/ECG_Resources_for_Reviewers.php
  • Resources Cochrane Collaboration: