Meta analysis and systematic reviews in qualitative research mission impossible
Download
1 / 32

Meta-analysis and systematic reviews in qualitative research: Mission impossible? - PowerPoint PPT Presentation


  • 98 Views
  • Uploaded on

Meta-analysis and systematic reviews in qualitative research: Mission impossible?. Ellen MacEachen (Institute for Work & Health, University of Toronto) Scott Reeves (Li Ka Shing Knowledge Institute, University of Toronto). Qualitative Inquiry Group Seminar

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Meta-analysis and systematic reviews in qualitative research: Mission impossible?' - trygg


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Meta analysis and systematic reviews in qualitative research mission impossible

Meta-analysis and systematic reviews in qualitative research: Mission impossible?

Ellen MacEachen (Institute for Work & Health, University of Toronto)

Scott Reeves (Li Ka Shing Knowledge Institute, University of Toronto)

Qualitative Inquiry Group Seminar

University of Toronto, March 27, 2009


Overview 4 parts
Overview – 4 parts research: Mission impossible?

1. Where did systematic reviews come from & why do we do them?

2. Brief overview of the systematic review process

3. Cases that draw out systematic review issues

  • Case I (Scott)

  • Case II (Ellen)

    4. Reflections on the conduct and usefulness of systematic reviews of qualitative studies


1 what are systematic reviews
1. What are systematic reviews? research: Mission impossible?

  • Positivistic roots / standardization

  • Synthesis of primary studies

  • Multiple check points / dual reviewers / transparency

  • Assess quality of evidence (only include ‘rigorous’ studies)

  • Inference through statistical analysis

  • Newer Qual types (meta-ethnography… )


Why do srs
Why do SRs? research: Mission impossible?

Useful:

  • Knowledge translation (policymakers/ practitioners)

  • Initial entry into field (academics)

  • Help define field (academics)

  • Outline areas of future research (acad/policy)

  • Grant applications (academics)


2 brief overview of the systematic review process
2. Brief overview of the research: Mission impossible?systematic review process

A walk through the ‘recipe’ followed for systematic reviews


What keywords are used in the search? research: Mission impossible?

What is the scope of the search? Languages? Dates?

What stakeholders are involved and at what stages?

What databases will be searched?

Medline

Total = 1620

EMBASE

Total = 1683

CINAHL

Total = 443

PsycINFO

Total = 1174

Sociological Abstracts Total = 279

ASSIA

Total = 74

ABI Inform

Total = 381

EconLit

Total = 58

Business Source Premier Total = 585

Content Experts’ Refs Total = 840

How will we find papers not listed on databases?

Other

Total = 240

Quality control process for what is being included/excluded

Step 1: Library Search

What articles are included/excluded?

Step 2: Study Relevance

Excluded at this level

n = 4256

Studies considered for QA (qualitative, quantitative and mixed methods

(n =609 )

Supplemental articles from reference list

n = 63

Two reviewers per paper, consensus approach

Step 3: Division of QNT/QL Studies

Excluded at this level

n = 20

Excluded at this level

n = 13

How will we assess quality?

Two reviewers per paper at this stage, consensus approach

Step 4: Quality Appraisal

What data will we extract?

Studies passing quality appraisaln = 5

Step 5: Data Extraction

Studies passing quality appraisal n = 14

Step 6: Finding Synthesis

n = 19


3 sr cases
3. SR ‘cases’ research: Mission impossible?

Case I – Making decisions about how to construct systematic reviews: the early days

Interprofessional education (Scott)

Case II – Struggles with the paradigm of systematic reviews: later days

Work & health in small businesses (Ellen)


Interprofessional education ipe
Interprofessional education (IPE) research: Mission impossible?

Case I: Making

decisions about how

to construct

systematic reviews:

the early days


SETTING PARAMETERS research: Mission impossible?

Discussion/consultation/agreement: definitions, approaches, processes

REVIEW QUESTION

what kind of IPE, under what circumstances, produces what kind of outcomes?

LITERATURE SEARCHES

Retrieval & screening

Quantitative studies:

Quality assessment, data extraction, synthesis

SYNTHESIS

Different types of IPE and associated outcomes

Qualitative studies:

Quality assessment, data extraction, synthesis

Mixed method studies:

Quality assessment, data extraction, synthesis


Context information research: Mission impossible?

  • Policymaker demands for IPE evidence (early 1990s)

  • SR – political response

  • In dark about SR process – new activity

  • ‘Side line’ work (enthusiasm)


SR processes research: Mission impossible?

  • Inclusive approach to review team – 9 members

  • Conceptually inclusive (implicit/explicit IPE)

  • Methodologically inclusive (qual/quan)

  • Open stance – seeking guidelines/standards

  • Pragmatic – trial & error

  • Abstraction – pre-determined categories


Thinking about Quality research: Mission impossible?

quality of study (appropriate design, sampling, recruitment, validity, reliability)

quality of information (good contextual info, explicit rationale, clear research questions, clear results)

(e.g. CASP, EPPI Centre, Popay et al 1998)


Large and small SRs research: Mission impossible?

  • Big was good: 107 studies

    Practitioners/policymakers - larger numbers (width)

  • Small was better - 21 studies

    Researchers – small numbers (higher quality)


Some reflections research: Mission impossible?

  • Lots of discussion/debate create parameters – first IPE review

  • Proceed with caution

  • One step forward, two back

  • Gradual movement: from inclusion to exclusion


Some reflections research: Mission impossible?

  • Different types of qualitative research difficult to synthesize

  • Qual / Quan / mixed methods – more problems

  • Quality assessment – best effort (pragmatics)

  • SR team dynamics


Case ii systematic review on work and health in small businesses ellen
Case II: Systematic review on work and health in small businesses (Ellen)

  • Struggles with the paradigm of systematic reviews

  • 2 parts:

    • qualitative review

    • mixed method review


REVIEW QUESTION businesses (Ellen)

What understandings, processes and interventions influence OHS in SBs?

SUB-QUESTIONS

Qualitative literature:

How do SB workplace parties understand and enact processes related toOHS?

Quantitative literature:

Do OHS interventions in SBs have an effect on OHS?

STAKEHOLDER CONSULTATION

LITERATURE SEARCH

Retrieval, screening of T&As

FOCI FOR IN-DEPTH REVIEW

Qualitative studies:

Quality assessment, data extraction, synthesis

Quantitative studies:

Quality assessment, data extraction, synthesis

STAKEHOLDER CONSULTATION

OVERALL SYNTHESIS

Understandings, processes and interventions that influence OHS in SBs


Struggles in a qualitative review
Struggles in a Qualitative review businesses (Ellen)

  • How to observe the broad focus that can be taken with qualitative research while also subjecting it to the ‘recipe’ of a SR process

    • The SR process is systematic, not exploratory.

    • Pre-defined parameters


Struggles in a qualitative review1
Struggles in a Qualitative review businesses (Ellen)

  • How to synthesize the findings & insights of Qualitative studies while also preserving their context and theory

    • The systematic review ‘extracts’ findings to answer a specific question.

    • What insights do we end up with? What do we gain/lose?


Struggles in a qualitative review2
Struggles in a Qualitative review businesses (Ellen)

  • What is being counted/considered in a qualitative systematic review?

  • Varying approaches to data extraction.

    • pre-set-findings categories?

    • grounded theory process?


Struggles in a qualitative review3
Struggles in a Qualitative review businesses (Ellen)

  • What counts as a qualitative study?

  • Studies can self-identify as “qualitative” but not use recognizable qualitative methods.

    • E.g. Participatory studies

  • Our criteria: Some qualitative data & some qualitative analysis of it


Struggles in a qualitative review4
Struggles in a Qualitative review businesses (Ellen)

  • How to handle interesting data from studies that don’t meet quality criteria?

    • “Nuggets” Pawson (2006).

    • Descriptions of process that are a ‘nugget of gold’ for the question being asked.


Struggles in a mixed method review
Struggles in a Mixed Method review businesses (Ellen)

  • How to achieve a steady process for both Qualitative & Quantitative sides of the review

  • Quantitative team-- focused on definitions, outcomes

  • Qualitative team--no fixed categories for outcomes.

  • Qual & quan work happens at different times--affects possibilities for concurrent synergy.


Struggles in a mixed method review1
Struggles in a Mixed Method review businesses (Ellen)

  • Theoretical differences about ‘conflict of interest’ during the quality assessment process

  • Qualitative team--sensitive to social/power relations

    • heated issue, agreed to disagree.


Struggles in a mixed method review2
Struggles in a Mixed Method review businesses (Ellen)

  • Differences about ‘levels of evidence’

  • QN additive approach

    • E.g. "best evidence synthesis guidelines”.

    • “How much evidence is there that this has an effect?”

  • QL studies—not about effectiveness

    • Some additive approach; e.g. concepts in 3 studies = theme

    • No overall conclusions about strength of evidence


Iv reflections on the conduct and usefulness of systematic reviews
IV. Reflections on the conduct and usefulness of systematic reviews

  • SR process developed from Quantitative paradigm; we are trying to adapt Qualitative literature to it…

    …Can this be meaningfully done?


Issues
Issues reviews

Reviews are not entirely systematic

  • Messiness

  • Paired reviewers: anti-bias ‘ideal’

  • Politics of agreement/consensus

    BUT:

  • Power relations among reviewers

  • Assessing papers: own merits or against overall standard?

  • QA approaches change over course of the review


Issues1
Issues reviews

  • The issue of the podium

    • Arrogance: decisions on inclusion of peer reviewed papers

  • A side effect of systematic reviews

    • Claims of ‘ownership’ of a field from SR work


Issues2
Issues reviews

  • Debates:

    • Can the synthesis integrate research using different methods/theories

    • Although there are multiple descriptions/ explanations of data, these all ultimately relate to some underlying reality/truth

      (Bondas & Hall 2007, Mays et al 2005)


Some gains
Some gains reviews

  • Think carefully about papers

  • Learn a lot about a field

  • Read a lot of poor papers (learn: good, bad ugly)

  • Hone critical appraisal skills (teaching, journal reviewing/editing)

  • Opportunities to engage with people (outside academia)


Thank you
Thank you reviews

  • Questions? Comments?


ad