Issues and Challenges with synthesis of qualitative evidence

1. Issues and Challenges with synthesis of qualitative evidence Professor Jane Noyes Director, Centre for Health-Related Research Bangor University Convenor Cochrane Qualitative Research Methods Group Editor Journal of Advanced Nursing

2. Some issues to consider 1. Protocol - How and when to select a synthesis approach? 3. Use of conceptual/theoretical frameworks to inform synthesis 4. Organisation and management of data for synthesis 5. What challenges are people reporting with specific approaches? 6. What methodological work still needs to be done?

3. Protocol - How and when to select a synthesis approach? • Upfront in protocol? • Once pool of evidence is known?

5. How best to organise and manage qualitative data ? Example of two WHO/Cochrane reviews

6. WHO Alliance commissioned reviews Evidence Synthesis on Strategies for Implementing Interventions for Achieving MDGs 4, 5 or 6 2010- 2011 Implementation Research Platform Our reviews also linked with GREAT Initiative

7. Guideline development • Research priorities • Evidence synthesis • Applicability of evidence • Transfer of knowledge

8. Primary objective of our work • To Optimize the delivery of key interventions to attain MDGs 4 & 5 (Optimize4MNH) through task-shifting • To identify effective practices that can be delivered by a lower cadre of health worker that would increase access to effective interventions in a safe way • To identify and make recommendations on cadre(s) of workers who can increase access to effective practices in a safe way

9. WHO Alliance commissioned reviews - linked to existing Cochrane effectiveness reviews Task-shifting to improve access to health care providers for MCH: Systematic review of qualitative research on barriers and facilitators to the implementation of lay health worker programmes 2. Doctor/Nurse substitutionto improve access to health care providers for MCH: Systematic review of qualitative research on barriers and facilitators to the implementation of nurse-led care

10. Experienced multi-skilled team (plus research officers) • Dr Claire Glenton- Knowledge Scientist (NKC) – Cochrane EPOC • Dr Simon Lewin- medic and SR - NKC, EPOC and South African Cochrane Centre – EPOC • Dr Arash Rashidian – GP and mixed method HSR researcher (BU, Tehran University, Iran) • Dr Chris Colvin - PhD cultural anthropology MPH in epidemiology and biostatistics. – South Africa • Dr Metin GülmezogluWHO/World Bank Special Programme of Research, Development and Research Training in Human Reproduction – Who Geneva/Cochrane. • Prof Jane Noyes – health services researcher – Cochrane qualitative research methods group BU and Cochrane. • Online virtual global community of practice (1500 members) • Regular face to face meetings with WHO experts (x 2 days in May 2011)

11. Organising and managing evidence for synthesis • Early decision in protocol development – lump or split question at question development stage (PICO or SPICE) • Split usually equates to more focussed conceptually secure review question We had no choice to revise scope: • What to do if you cannot split/reduce review and it remains a large unmanageable ‘lump’?

12. Developing a Typology to support synthesis of evidence • Doctor/Nurse substitution • Developed/developing or developed country context • High medium or low complexity re substitution/task • Close or distant proximity of medical support • High/low training/preparation of nurse • Context of care delivery (team/alone/available resources/rural/urban/in-hours/out of hours)

13. Model of implementation to aid interpretation of barriers and facilitators evidence Realist approach: Develop an a priori theory/propositions of what works: • Context • Mechanism • Outcome Or model of implementation such as the PARiHS Framework map data against context, evidence, facilitation concepts

14. Logic models We are looking at the use of logic models in Cochrane For examples and instruction on developing a Logic Model see: University of Wisconsin - Extension, Cooperative Extension, Program Development and Evaluation http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

15. Management of evidence for synthesis Issues and challenges: Nurse doctor substitution – 50 studies Lay health worker – 100s of studies Do we sample – if so how? What criteria do we use? High/low quality evidence? High/low coverage of interventions? High/low coverage of context? Challenge is that low quality, low coverage of context and intervention usually occur together

16. The GRADE approach Grades of Recommendation Assessment, Development and Evaluation www.GradeWorking-Group.org

17. Quality of evidence across studies 1B I B II V III I A IV GRADE Outcome # 2 Outcome # 1 Outcome # 3 Quality: High Quality: Low Quality: Moderate 17

18. Reporting the synthesis Depends largely on approach to synthesis. Narrative – themes New theory - model Issues: Reporting quotes form original studies Themes derived from a single study

20. Challenges experienced by authors Meta-ethnography – Lewin et al Realist synthesis – Rycroft-Malone et al Thematic analysis – Thomas and Harden Complex interventions – Noyes et al

21. “I think you should be more explicit here in Step Two.”

22. Methodological work CQRMG Testing Lewin, Oxman and Glenton’s tool to assess the complexity of interventions , Developing guidance on when it may be appropriate to use a conceptual model/framework/theory to inform understanding and interpretation of complex evidence and effects. Determining: 3. When qualitative research should be included alongside a review of the effects of a complex intervention 4. How to structure qualitative reviews so as to facilitate the integration of their findings 5. How to structure effects reviews so as to facilitate the integration of their findings 6. How quality appraisal/assessment of bias should be addressed and reported in hybrid reviews 7. Methods for integrating and using qualitative data to understand complexity 8. How to present the results of a review that has integrated effects and qualitative data, for instance in a Summary of Findings table

23. Your thoughts?Open discussion and questions