Mixed method reviews
1 / 30

Mixed method reviews - PowerPoint PPT Presentation

  • Uploaded on

Mixed method reviews. Data analysis and synthesis. Complexity. “Some problems are so complex that you have to be highly intelligent just to be undecided about them” Lawrence J. Peter. Narrative synthesis (NS).

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Mixed method reviews' - jud

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Mixed method reviews

Mixed method reviews

Data analysis and synthesis


“Some problems are so complex that you have to be highly intelligent just to be undecided about them”

Lawrence J. Peter

Narrative synthesis ns
Narrative synthesis (NS)

  • An approach to the synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis

  • Designed for integrating effectiveness and implementation data, but can be used to synthesise evidence for a wide range of questions including effectiveness, experience, need, process, implementation etc.

Using a synthesis method


Assists in review team communication and documentation

Assists in making the processes of synthesis available to others for validation

Assists with the presentation and communication of the results of synthesis

Using a synthesis method


  • Pope C, Mays N, Popay J (2007) Synthesising Quantitative and Qualitative Evidence: a guide to methods. Milton Keynes,Open University Press

  • Popay J et al. (2006) Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. Available at:


What ns is and isn t

A framework to increase the transparency and trustworthiness of the synthesis

Includes description of tools and techniques

Could be used to synthesise the findings from mixed evidence sources

A guide to the whole process of systematic review – only the stage of synthesis

A cookbook of tools and techniques to be selected at random

Aims to be accessible, but not a guide for complete novices in research

What NS is, and isn’t

Relationship to systematic review
Relationship to systematic review trustworthiness of the synthesis

Question development

Theoretical framework

Search/data extraction

Quality assessment

Descriptive analysis

Preliminary synthesis

Interpretive analysis

Final modelling/synthesis

Theory testing /development

Iterative not linear

Looking for


in data

Generalise on basis of theory

Phases of ns
Phases of NS trustworthiness of the synthesis

  • Phase 1: Develop a theoretical framework*

  • Phase 2: Descriptive analysis + develop a preliminary synthesis

  • Phase 3: Interpretive analysis: explore relationships in the data + construct synthesis

  • Phase 4: Assess the robustness of the synthesis

  • NB Not linear stages: phases are iterative

    *not always required

Theoretical framework
Theoretical framework trustworthiness of the synthesis

  • Purpose: to inform decisions about the review question, contribute to interpretation of findings, and to assess the applicability of the findings

  • Sources:

    • Research question

    • Topic

    • Theory

    • Hypotheses

Descriptive analysis
Descriptive analysis trustworthiness of the synthesis

  • Purpose: to organise findings to provide an initial description of patterns across the included studies

  • Tasks: select, group, juxtapose

  • Tools:

    • Textual descriptions

    • Groupings and clusters

    • Tabulation

    • Transform to a common rubric

    • Vote count

    • Thematic analysis

    • Content analysis

Descriptive analysis1
Descriptive analysis trustworthiness of the synthesis

Descriptive analysis2
Descriptive analysis trustworthiness of the synthesis




Descriptive analysis3
Descriptive analysis trustworthiness of the synthesis

  • Textual descriptions

  • Groupings and clusters

  • Tabulation

  • Transform to a common rubric

  • Counting

  • Thematic analysis

  • Content analysis

Tools for describing examples
Tools for describing: examples trustworthiness of the synthesis

  • Textual descriptions: a short ‘story’ with key messages, significant information that acts as a ‘viewing platform’.

  • Ordering/grouping: finding shared features and clustering studies along those lines

  • Tabulation: Tables summarising key features of study objectives, design, context, findings

  • Common rubric: Using a common framework to allocate material from multiple studies

  • Counting: Using occurrence or vote counting to consider the frequency of phenomena in the data

  • Content/thematic analysis: qualitative analysis

Exercise description
EXERCISE: description trustworthiness of the synthesis

  • Select summary information from the narrative description to put in a table, making sure that the table will fit data from all three studies

  • Look at the types of clients, and context. Can you spot potential groupings?

  • Look at the findings across the three studies. Can you spot potential categories or themes?

Output preliminary synthesis
Output: preliminary synthesis trustworthiness of the synthesis

  • Descriptive narrative and tabular summary of the studies and findings

  • Specification of analytical framework/common rubric

    • Events, participants, contexts?

    • Stages, phases, time?

    • Ordering, grouping?

    • Positives, negatives?

    • Hierarchies, key issues?

  • Preliminary synthesis

    • Testing the theory with a subset of data to make sure it works

Exploring relationships in the data
Exploring relationships in the data trustworthiness of the synthesis

  • Purpose: to consider relationships between:

    • study results and key aspects of those studies (within)

    • factors across included studies (between)

  • Tasks: compare, combine, translate, integrate

  • Tools:

    • Identifying mediators and moderators

    • Sub-set analysis

    • Conceptual mapping

    • Translation: reciprocal/refutational

    • Qualitative case description

Exploring relationships in the data1
Exploring relationships in the data trustworthiness of the synthesis







Exploring relationships in the data2
Exploring relationships in the data trustworthiness of the synthesis

Mediators and moderators

Sub-set analysis

Conceptual mapping

Translation: reciprocal/ refutational

Qualitative case descriptions



Exploring relationships tools
Exploring relationships: tools trustworthiness of the synthesis

  • Moderators and mediators: categorising and viewing the relationship between concepts according to their action on the phenomena of interest

  • Sub-set analysis: Looking at actions, results or themes in groups, dependent on characteristics

  • Visual representation:

    • Concept mapping: links between concepts are labelled and placed in space to express a relationship

    • Ideas webbing: concepts organised in trees, hierarchies

    • Causal/cognitive mapping: representation aims to describe direction of influences, key issues in real life

  • Translation: reciprocal/refutational: grouping and integrating study findings in terms of level of agreement/fit in argument/conclusions

  • Qualitative case description: writing a short narrative summary across studies, concepts, themes, arguments

Output synthesis
Output: Synthesis trustworthiness of the synthesis

  • Rigorous exploration of alternative interpretations of relationships between and across studies

  • Some level of translation and integration with explanation of (e.g.)

    • agreement/refutation

    • commonalities/outliers

    • order/groupings

    • priorities/emphasis/frequency

  • Identification of moderators (influencing factors), mediators (mechanisms of action) and sub-groups (components of programme theory)

Exercise interpretation
EXERCISE: interpretation trustworthiness of the synthesis

  • Look at what you think might be the key concepts/themes drawn from the findings of the 3 studies

  • Play with the concepts/themes spatially to explore the potential relationships and patterns across concepts

  • Map the concepts/themes ACROSS STUDIES to denote some aspect of the relationship between them

Assessing robustness
Assessing robustness trustworthiness of the synthesis

  • Purpose: to assess the strength of evidence for drawing conclusions about the results of the synthesis in relation to methodological quality, relevance, trustworthiness

  • Tasks: assess relationship of synthesis to original review question, programme theory, study quality and review method

  • Tools:

    • Assessing theoretical fit

    • Validity assessment

    • Weight of evidence

    • Critical reflection

    • Triangulation

    • Checking with authors

Assessing robustness1
Assessing robustness trustworthiness of the synthesis



Assessing robustness2
Assessing robustness trustworthiness of the synthesis


Best Evidence Synthesis

Weight of evidence

Validity assessment

Critical reflection

Checking with authors





Tools assessing robustness
Tools: assessing robustness trustworthiness of the synthesis

  • Best evidence synthesis: prioritising/choosing studies with the strongest design

  • Weight of evidence: using a criterion of strength or value to interpret the synthesis findings e.g. effect size, repeatability, harmony, coherence

  • Use of validity/credibility assessment: using the results of critical appraisal to interpret synthesis

  • Critical reflection: using the review author’s use of critical tools/techniques/methods to consider results e.g. agreement, alternative explanations, assessment of theoretical fit, parsimony, elegance

  • Checking with authors: involving the authors of the primary studies in comment, reflection, or building theory/synthesis

  • Triangulation: using the results of other data sources to compare the results of the synthesis

Exercise trustworthiness of the synthesis

  • How do you think the methods and quality of the original studies would affect the interpretation of the findings?

Output robustness
Output: robustness trustworthiness of the synthesis

  • Identification of any limitations in the process of review and synthesis

  • Assessment of the credibility and trustworthiness of the synthesis product, seen through the provenance, quality, and strength of the evidence

  • Assessment of the significance of the theoretical contribution

  • Identification of its potential use and further development

Review product suggestions
Review product suggestions trustworthiness of the synthesis

  • Formal report: PRISMA guidelines

    • method

    • analysis: description, interpretation

    • conclusion: theoretical contribution

    • implications

  • Practitioner report:

    • content,

    • examples, visual illustration

    • best practice guidance

    • tools, resource lists

Common problems
Common problems trustworthiness of the synthesis

  • Volume of material

  • Level of material

  • Heterogeneity of material

  • Disaggregation

  • Crossing disciplinary boundaries

  • Lack of guidance on synthesis method

  • Time

  • Skill in original research methodology