Mixed method reviews
Download
1 / 30

Mixed method reviews - PowerPoint PPT Presentation


  • 293 Views
  • Uploaded on

Mixed method reviews. Data analysis and synthesis. Complexity. “Some problems are so complex that you have to be highly intelligent just to be undecided about them” Lawrence J. Peter. Narrative synthesis (NS).

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Mixed method reviews' - jud


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Mixed method reviews

Mixed method reviews

Data analysis and synthesis


Complexity
Complexity

“Some problems are so complex that you have to be highly intelligent just to be undecided about them”

Lawrence J. Peter


Narrative synthesis ns
Narrative synthesis (NS)

  • An approach to the synthesis of findings from multiple studies that relies primarily on the use of words and text to summarise and explain the findings of the synthesis

  • Designed for integrating effectiveness and implementation data, but can be used to synthesise evidence for a wide range of questions including effectiveness, experience, need, process, implementation etc.


Using a synthesis method

Advantages

Assists in review team communication and documentation

Assists in making the processes of synthesis available to others for validation

Assists with the presentation and communication of the results of synthesis

Using a synthesis method


Guidance
Guidance

  • Pope C, Mays N, Popay J (2007) Synthesising Quantitative and Qualitative Evidence: a guide to methods. Milton Keynes,Open University Press

  • Popay J et al. (2006) Guidance on the Conduct of Narrative Synthesis in Systematic Reviews. Available at:

    http://www.lancs.ac.uk/shm/research/nssr/research/dissemination/publications/NS_Synthesis_Guidance_v1.pdf


What ns is and isn t

A framework to increase the transparency and trustworthiness of the synthesis

Includes description of tools and techniques

Could be used to synthesise the findings from mixed evidence sources

A guide to the whole process of systematic review – only the stage of synthesis

A cookbook of tools and techniques to be selected at random

Aims to be accessible, but not a guide for complete novices in research

What NS is, and isn’t


Relationship to systematic review
Relationship to systematic review trustworthiness of the synthesis

Question development

Theoretical framework

Search/data extraction

Quality assessment

Descriptive analysis

Preliminary synthesis

Interpretive analysis

Final modelling/synthesis

Theory testing /development

Iterative not linear

Looking for

patterns

in data

Generalise on basis of theory


Phases of ns
Phases of NS trustworthiness of the synthesis

  • Phase 1: Develop a theoretical framework*

  • Phase 2: Descriptive analysis + develop a preliminary synthesis

  • Phase 3: Interpretive analysis: explore relationships in the data + construct synthesis

  • Phase 4: Assess the robustness of the synthesis

  • NB Not linear stages: phases are iterative

    *not always required


Theoretical framework
Theoretical framework trustworthiness of the synthesis

  • Purpose: to inform decisions about the review question, contribute to interpretation of findings, and to assess the applicability of the findings

  • Sources:

    • Research question

    • Topic

    • Theory

    • Hypotheses


Descriptive analysis
Descriptive analysis trustworthiness of the synthesis

  • Purpose: to organise findings to provide an initial description of patterns across the included studies

  • Tasks: select, group, juxtapose

  • Tools:

    • Textual descriptions

    • Groupings and clusters

    • Tabulation

    • Transform to a common rubric

    • Vote count

    • Thematic analysis

    • Content analysis


Descriptive analysis1
Descriptive analysis trustworthiness of the synthesis


Descriptive analysis2
Descriptive analysis trustworthiness of the synthesis

SELECT

GROUP

JUXTAPOSE


Descriptive analysis3
Descriptive analysis trustworthiness of the synthesis

  • Textual descriptions

  • Groupings and clusters

  • Tabulation

  • Transform to a common rubric

  • Counting

  • Thematic analysis

  • Content analysis


Tools for describing examples
Tools for describing: examples trustworthiness of the synthesis

  • Textual descriptions: a short ‘story’ with key messages, significant information that acts as a ‘viewing platform’.

  • Ordering/grouping: finding shared features and clustering studies along those lines

  • Tabulation: Tables summarising key features of study objectives, design, context, findings

  • Common rubric: Using a common framework to allocate material from multiple studies

  • Counting: Using occurrence or vote counting to consider the frequency of phenomena in the data

  • Content/thematic analysis: qualitative analysis


Exercise description
EXERCISE: description trustworthiness of the synthesis

  • Select summary information from the narrative description to put in a table, making sure that the table will fit data from all three studies

  • Look at the types of clients, and context. Can you spot potential groupings?

  • Look at the findings across the three studies. Can you spot potential categories or themes?


Output preliminary synthesis
Output: preliminary synthesis trustworthiness of the synthesis

  • Descriptive narrative and tabular summary of the studies and findings

  • Specification of analytical framework/common rubric

    • Events, participants, contexts?

    • Stages, phases, time?

    • Ordering, grouping?

    • Positives, negatives?

    • Hierarchies, key issues?

  • Preliminary synthesis

    • Testing the theory with a subset of data to make sure it works


Exploring relationships in the data
Exploring relationships in the data trustworthiness of the synthesis

  • Purpose: to consider relationships between:

    • study results and key aspects of those studies (within)

    • factors across included studies (between)

  • Tasks: compare, combine, translate, integrate

  • Tools:

    • Identifying mediators and moderators

    • Sub-set analysis

    • Conceptual mapping

    • Translation: reciprocal/refutational

    • Qualitative case description


Exploring relationships in the data1
Exploring relationships in the data trustworthiness of the synthesis

COMPARE

COMBINE

TRANSLATE

INTEGRATE

RELATIONSHIPS ACROSS STUDIES

RELATIONSHIPSWITHIN STUDIES


Exploring relationships in the data2
Exploring relationships in the data trustworthiness of the synthesis

Mediators and moderators

Sub-set analysis

Conceptual mapping

Translation: reciprocal/ refutational

Qualitative case descriptions

RELATIONSHIPS ACROSS STUDIES

RELATIONSHIPS WITHIN STUDIES


Exploring relationships tools
Exploring relationships: tools trustworthiness of the synthesis

  • Moderators and mediators: categorising and viewing the relationship between concepts according to their action on the phenomena of interest

  • Sub-set analysis: Looking at actions, results or themes in groups, dependent on characteristics

  • Visual representation:

    • Concept mapping: links between concepts are labelled and placed in space to express a relationship

    • Ideas webbing: concepts organised in trees, hierarchies

    • Causal/cognitive mapping: representation aims to describe direction of influences, key issues in real life

  • Translation: reciprocal/refutational: grouping and integrating study findings in terms of level of agreement/fit in argument/conclusions

  • Qualitative case description: writing a short narrative summary across studies, concepts, themes, arguments


Output synthesis
Output: Synthesis trustworthiness of the synthesis

  • Rigorous exploration of alternative interpretations of relationships between and across studies

  • Some level of translation and integration with explanation of (e.g.)

    • agreement/refutation

    • commonalities/outliers

    • order/groupings

    • priorities/emphasis/frequency

  • Identification of moderators (influencing factors), mediators (mechanisms of action) and sub-groups (components of programme theory)


Exercise interpretation
EXERCISE: interpretation trustworthiness of the synthesis

  • Look at what you think might be the key concepts/themes drawn from the findings of the 3 studies

  • Play with the concepts/themes spatially to explore the potential relationships and patterns across concepts

  • Map the concepts/themes ACROSS STUDIES to denote some aspect of the relationship between them


Assessing robustness
Assessing robustness trustworthiness of the synthesis

  • Purpose: to assess the strength of evidence for drawing conclusions about the results of the synthesis in relation to methodological quality, relevance, trustworthiness

  • Tasks: assess relationship of synthesis to original review question, programme theory, study quality and review method

  • Tools:

    • Assessing theoretical fit

    • Validity assessment

    • Weight of evidence

    • Critical reflection

    • Triangulation

    • Checking with authors


Assessing robustness1
Assessing robustness trustworthiness of the synthesis

REVIEW QUESTIONS

PROGRAMME THEORY


Assessing robustness2
Assessing robustness trustworthiness of the synthesis

REVIEW QUESTIONS

Best Evidence Synthesis

Weight of evidence

Validity assessment

Critical reflection

Checking with authors

Triangulation

STUDY QUALITY

REVIEW METHOD

PROGRAMME THEORY


Tools assessing robustness
Tools: assessing robustness trustworthiness of the synthesis

  • Best evidence synthesis: prioritising/choosing studies with the strongest design

  • Weight of evidence: using a criterion of strength or value to interpret the synthesis findings e.g. effect size, repeatability, harmony, coherence

  • Use of validity/credibility assessment: using the results of critical appraisal to interpret synthesis

  • Critical reflection: using the review author’s use of critical tools/techniques/methods to consider results e.g. agreement, alternative explanations, assessment of theoretical fit, parsimony, elegance

  • Checking with authors: involving the authors of the primary studies in comment, reflection, or building theory/synthesis

  • Triangulation: using the results of other data sources to compare the results of the synthesis


Exercise
Exercise trustworthiness of the synthesis

  • How do you think the methods and quality of the original studies would affect the interpretation of the findings?


Output robustness
Output: robustness trustworthiness of the synthesis

  • Identification of any limitations in the process of review and synthesis

  • Assessment of the credibility and trustworthiness of the synthesis product, seen through the provenance, quality, and strength of the evidence

  • Assessment of the significance of the theoretical contribution

  • Identification of its potential use and further development


Review product suggestions
Review product suggestions trustworthiness of the synthesis

  • Formal report: PRISMA guidelines

    • method

    • analysis: description, interpretation

    • conclusion: theoretical contribution

    • implications

  • Practitioner report:

    • content,

    • examples, visual illustration

    • best practice guidance

    • tools, resource lists


Common problems
Common problems trustworthiness of the synthesis

  • Volume of material

  • Level of material

  • Heterogeneity of material

  • Disaggregation

  • Crossing disciplinary boundaries

  • Lack of guidance on synthesis method

  • Time

  • Skill in original research methodology