Mixed method reviews - PowerPoint PPT Presentation

Mixed method reviews
1 / 30

  • Uploaded on
  • Presentation posted in: General

Mixed method reviews. Data analysis and synthesis. Complexity. “Some problems are so complex that you have to be highly intelligent just to be undecided about them” Lawrence J. Peter. Narrative synthesis (NS).

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Mixed method reviews

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Mixed method reviews

Mixed method reviews

Data analysis and synthesis



“Some problems are so complex that you have to be highly intelligent just to be undecided about them”

Lawrence J. Peter

Narrative synthesis ns

Narrative synthesis (NS)

  • An approach to the synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis

  • Designed for integrating effectiveness and implementation data, but can be used to synthesise evidence for a wide range of questions including effectiveness, experience, need, process, implementation etc.

Using a synthesis method


Assists in review team communication and documentation

Assists in making the processes of synthesis available to others for validation

Assists with the presentation and communication of the results of synthesis

Using a synthesis method



  • Pope C, Mays N, Popay J (2007) Synthesising Quantitative and Qualitative Evidence: a guide to methods. Milton Keynes,Open University Press

  • Popay J et al. (2006) Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. Available at:


What ns is and isn t

A framework to increase the transparency and trustworthiness of the synthesis

Includes description of tools and techniques

Could be used to synthesise the findings from mixed evidence sources

A guide to the whole process of systematic review – only the stage of synthesis

A cookbook of tools and techniques to be selected at random

Aims to be accessible, but not a guide for complete novices in research

What NS is, and isn’t

Relationship to systematic review

Relationship to systematic review

Question development

Theoretical framework

Search/data extraction

Quality assessment

Descriptive analysis

Preliminary synthesis

Interpretive analysis

Final modelling/synthesis

Theory testing /development

Iterative not linear

Looking for


in data

Generalise on basis of theory

Phases of ns

Phases of NS

  • Phase 1: Develop a theoretical framework*

  • Phase 2: Descriptive analysis + develop a preliminary synthesis

  • Phase 3: Interpretive analysis: explore relationships in the data + construct synthesis

  • Phase 4: Assess the robustness of the synthesis

  • NB Not linear stages: phases are iterative

    *not always required

Theoretical framework

Theoretical framework

  • Purpose: to inform decisions about the review question, contribute to interpretation of findings, and to assess the applicability of the findings

  • Sources:

    • Research question

    • Topic

    • Theory

    • Hypotheses

Descriptive analysis

Descriptive analysis

  • Purpose: to organise findings to provide an initial description of patterns across the included studies

  • Tasks: select, group, juxtapose

  • Tools:

    • Textual descriptions

    • Groupings and clusters

    • Tabulation

    • Transform to a common rubric

    • Vote count

    • Thematic analysis

    • Content analysis

Descriptive analysis1

Descriptive analysis

Descriptive analysis2

Descriptive analysis




Descriptive analysis3

Descriptive analysis

  • Textual descriptions

  • Groupings and clusters

  • Tabulation

  • Transform to a common rubric

  • Counting

  • Thematic analysis

  • Content analysis

Tools for describing examples

Tools for describing: examples

  • Textual descriptions: a short ‘story’ with key messages, significant information that acts as a ‘viewing platform’.

  • Ordering/grouping: finding shared features and clustering studies along those lines

  • Tabulation: Tables summarising key features of study objectives, design, context, findings

  • Common rubric: Using a common framework to allocate material from multiple studies

  • Counting: Using occurrence or vote counting to consider the frequency of phenomena in the data

  • Content/thematic analysis: qualitative analysis

Exercise description

EXERCISE: description

  • Select summary information from the narrative description to put in a table, making sure that the table will fit data from all three studies

  • Look at the types of clients, and context. Can you spot potential groupings?

  • Look at the findings across the three studies. Can you spot potential categories or themes?

Output preliminary synthesis

Output: preliminary synthesis

  • Descriptive narrative and tabular summary of the studies and findings

  • Specification of analytical framework/common rubric

    • Events, participants, contexts?

    • Stages, phases, time?

    • Ordering, grouping?

    • Positives, negatives?

    • Hierarchies, key issues?

  • Preliminary synthesis

    • Testing the theory with a subset of data to make sure it works

Exploring relationships in the data

Exploring relationships in the data

  • Purpose: to consider relationships between:

    • study results and key aspects of those studies (within)

    • factors across included studies (between)

  • Tasks: compare, combine, translate, integrate

  • Tools:

    • Identifying mediators and moderators

    • Sub-set analysis

    • Conceptual mapping

    • Translation: reciprocal/refutational

    • Qualitative case description

Exploring relationships in the data1

Exploring relationships in the data







Exploring relationships in the data2

Exploring relationships in the data

Mediators and moderators

Sub-set analysis

Conceptual mapping

Translation: reciprocal/ refutational

Qualitative case descriptions



Exploring relationships tools

Exploring relationships: tools

  • Moderators and mediators: categorising and viewing the relationship between concepts according to their action on the phenomena of interest

  • Sub-set analysis: Looking at actions, results or themes in groups, dependent on characteristics

  • Visual representation:

    • Concept mapping: links between concepts are labelled and placed in space to express a relationship

    • Ideas webbing: concepts organised in trees, hierarchies

    • Causal/cognitive mapping: representation aims to describe direction of influences, key issues in real life

  • Translation: reciprocal/refutational: grouping and integrating study findings in terms of level of agreement/fit in argument/conclusions

  • Qualitative case description: writing a short narrative summary across studies, concepts, themes, arguments

Output synthesis

Output: Synthesis

  • Rigorous exploration of alternative interpretations of relationships between and across studies

  • Some level of translation and integration with explanation of (e.g.)

    • agreement/refutation

    • commonalities/outliers

    • order/groupings

    • priorities/emphasis/frequency

  • Identification of moderators (influencing factors), mediators (mechanisms of action) and sub-groups (components of programme theory)

Exercise interpretation

EXERCISE: interpretation

  • Look at what you think might be the key concepts/themes drawn from the findings of the 3 studies

  • Play with the concepts/themes spatially to explore the potential relationships and patterns across concepts

  • Map the concepts/themes ACROSS STUDIES to denote some aspect of the relationship between them

Assessing robustness

Assessing robustness

  • Purpose: to assess the strength of evidence for drawing conclusions about the results of the synthesis in relation to methodological quality, relevance, trustworthiness

  • Tasks: assess relationship of synthesis to original review question, programme theory, study quality and review method

  • Tools:

    • Assessing theoretical fit

    • Validity assessment

    • Weight of evidence

    • Critical reflection

    • Triangulation

    • Checking with authors

Assessing robustness1

Assessing robustness



Assessing robustness2

Assessing robustness


Best Evidence Synthesis

Weight of evidence

Validity assessment

Critical reflection

Checking with authors





Tools assessing robustness

Tools: assessing robustness

  • Best evidence synthesis: prioritising/choosing studies with the strongest design

  • Weight of evidence: using a criterion of strength or value to interpret the synthesis findings e.g. effect size, repeatability, harmony, coherence

  • Use of validity/credibility assessment: using the results of critical appraisal to interpret synthesis

  • Critical reflection: using the review author’s use of critical tools/techniques/methods to consider results e.g. agreement, alternative explanations, assessment of theoretical fit, parsimony, elegance

  • Checking with authors: involving the authors of the primary studies in comment, reflection, or building theory/synthesis

  • Triangulation: using the results of other data sources to compare the results of the synthesis



  • How do you think the methods and quality of the original studies would affect the interpretation of the findings?

Output robustness

Output: robustness

  • Identification of any limitations in the process of review and synthesis

  • Assessment of the credibility and trustworthiness of the synthesis product, seen through the provenance, quality, and strength of the evidence

  • Assessment of the significance of the theoretical contribution

  • Identification of its potential use and further development

Review product suggestions

Review product suggestions

  • Formal report: PRISMA guidelines

    • method

    • analysis: description, interpretation

    • conclusion: theoretical contribution

    • implications

  • Practitioner report:

    • content,

    • examples, visual illustration

    • best practice guidance

    • tools, resource lists

Common problems

Common problems

  • Volume of material

  • Level of material

  • Heterogeneity of material

  • Disaggregation

  • Crossing disciplinary boundaries

  • Lack of guidance on synthesis method

  • Time

  • Skill in original research methodology

  • Login