Conducting Systematic Reviews from a KT Perspective

1. Conducting Systematic Reviews from a KT Perspective Important information before we start… • Turn speakers off • Mute your phone when you are not speaking • Turn on the captioning

2. Conducting Systematic Reviews from a KT Perspective: Developing evidence-based information for researchers, clinicians and consumers Model Systems Knowledge Translation Center University of Washington

3. The Webcast Series • Part 1: Conducting Systematic Reviews from a KT Perspective: lessons learned from 14 systematic reviews collaboratively written with faculty from the Model Systems in TBI, SCI and Burn about conducting reviews from a KT perspective. • Part 2: The MSKTC Approach to Systematic Reviews: using tools and forms developed by the MSKTC to conduct systematic reviews of the evidence. • Part 3: Using the MSKTC Systematic Review database: a demonstration of a database with web interface to expedite the creation of tables of evidence. This database will be available to NIDRR grantees.

4. Using Elluminate Elluminate is an interactive, real-time, online system. • Interact with moderators • Raise hand • Laugh • Applaud • Show disapproval • Show confusion • Chat • Ask questions, offer comments • Using the Chat space, tell us which Model System you are associated with • Poll • Respond to quick polls or surveys

5. Quick Poll • Have you ever conducted a systematic review?

6. Goals for this Webcast • Conduct systematic reviews from a knowledge translation perspective • Learn about different types of and approaches to systematic reviews • Learn about conducting systematic reviews that result in information that is useful to multiple audiences, including clinicians and consumers • Learn about some of the major challenges in conducting systematic reviews in rehabilitation

7. What is Knowledge Translation? • An active process of ensuring that new knowledge gained through the course of research ultimately improves the lives of people with disabilities, and furthers their participation in society (NIDRR, 2005). • Process that takes us “from knowledge creation through implementation and impact.”(Sudsawad, 2007)

8. Current Barriers to KT Consumers: difficulty finding and evaluating research to guide health care and rehabilitation decision making. Health care professionals: lack time, resources, or skill to translate research to practice and rate evidence Researchers: academic culture does not support publication of non-refereed materials such as consumer education materials Policy makers: disconnect from researchers—want “sound bytes” rather than complex analysis Gov’t agencies: political or agency constraints may outweigh evidence in guiding research and policy agendas

9. What is a systematic review? A systematic review is an attempt to gather all empirical evidence that fits pre-specified eligibility criteria in order to answer a specific research or clinical question.   • Key characteristics: • A clearly stated set of objectives with pre-defined eligibility criteria; • An explicit, reproducible methodology; • A systematic search that attempts to identify all the studies that meet the eligibility criteria; • An assessment of the validity of the findings Source: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.0.1 [updated September 2008]. The Cochrane Collaboration, 2008. Available from www.cochrane-handbook.org.

10. Systematic Reviews from a KT Perspective • Create a framework • The systematic review is not the end product • It is one source of information • Other sources include professional judgment or consensus • What do you do when the evidence isn’t adequate? • Gather other best evidence to communicate to your audience, .e.g., consumers • Saying ‘there is no evidence’ is not a satisfactory or useful answer • Ensure the clinical relevance of the question • Use a pragmatic approach in conducting systematic reviews from a KT perspective

11. How does a Systematic Review differ from a Narrative or Lit Review? Adapted from: Cook, D., Mulrow, C. & Haynes, R. (1997). Systematic Reviews: Synthesis of best evidence for clinical decisions. Annals of Internal Medicine, 126(5), p.376-380).

12. What are the Most Common Types of Systematic Reviews? • Intervention • To assessthe evidence about the effects of a healthcare intervention. • Prognostic • To assess the evidence of models or predictors of patient outcomes. • Measurement • To assess the properties of health-status instruments or tools

13. Example #1: TBI & Depression Interventions(http://www.liebertonline.com/doi/pdfplus/10.1089/neu.2009.1091) Purpose: • critically evaluate the evidence on interventions for depression following TBI • provide recommendations for clinical practice & future research Outcome: • There is a paucity of RCTs for depression following TBI. • Serotonergic antidepressants and cognitive behavioral interventions appear to have the best preliminary evidence for treating depression following TBI. • More research is needed to provide evidence based treatment recommendations for depression following TBI

14. What is Evidence Based? • There are many types of evidence, e.g. • RCTs (large & small) • Nonrandomized trials • Case series • Expert consensus • It’s an active process • Evidence you will trust • Conveying the evidence

15. Issues in rehab systematic reviews • Reviews report finding very little or no evidence • Evidence standards & methods not a good match to rehab • Few RCTs in rehab • Johnston et al list multiple reasons, including….

16. Issues in Rehabilitation Research • Complexity of rehab interventions • Behavioral, psychosocial, environmental • Participatory research is highly valued but may be considered less rigorous • Small sample sizes; need for highly individualized interventions (e.g., AT) • Blinding & placebo control (again, AT is a good example) • Control group or denial of services? • Rehab perceived to be a clinical service not research & developments • Many of our important issues cannot be manipulated experimentally (attitudes, access, economic factors)

17. Steps in Conducting a Systematic Review from a KT Perspective

18. Develop a Clinical Question • To write a good clinical question, you should ask yourself • Why am I conducting the review? • What do I hope to learn? • Who is this information for? • Consumers, clinicians, researchers, others?

19. A good clinical question will…. • Save time when researching • Keep the focus directly on the needs of the target audience • Suggest the appropriate form that a useful answer may take

20. Clinical Question Which of the following questions would result in the most useful clinical findings for patients with SCI? • Is cardiovascular exercise beneficial to patients with SCI? • What are the adverse effects of cardiovascular exercise in individuals with SCI? • What are the parameters/components of a safe cardiovascular exercise program? 

21. Clinical Question Poll Response

22. Clinical Question—MSKTC Suggestions Which of the following questions would result in the most useful clinical findings for patients with SCI? • Is cardiovascular exercise beneficial to patients with SCI? • oversimplified, we already know this is not a Yes/No question • What are the adverse effects of cardiovascular exercise in individuals with SCI? • only focuses on the adverse effects, patients may also want to know what to do, not just what not to do • What are the parameters/components of a safe cardiovascular exercise program?  • BEST ONE!

23. Create a Plan Your development plan should include: • Your clinical question • Criteria for the literature search • Key words • Databases • Dates to include (how far back do you go?) • Inclusion and exclusion criteria • Types of studies you want to include • A timeline for your project

24. Key Words • Key Text words for the condition, e.g., • Spinal Cord Injury, Paraplegia, Tetraplegia, Quadriplegia • Key Text words for the intervention, e.g., • Training • Exercise conditioning • Activity limitation • FES (Functional Electrical Stimulation)

25. Criteria • Outcomes to be included… • Cardiovascular events, autonomic dysreflexia • And excluded • Beneficial effects of exercise • Types of studies to be included • RCTs, case control, case cohort

26. Study Timeline

27. Categorize • Create categories for the articles that you have found according to your inclusion and exclusion criteria. • Once you have categorized your articles, you can determine how many levels to include in your review

28. Deciding which Articles to Include Categories 1The primary outcome or one of the primary outcomes of the experimental study is related to maintaining cardiovascular health after SCI2 (NOT interventions specifically aimed to help gain movement or walk). These interventions could include upper body endurance exercise, strength & endurance training, etc.2The primary outcome or one of the primary outcomes of the observational study is related to maintaining cardiovascular health after SCI2 (NOT interventions specifically aimed to help gain movement or walk). 3A secondary outcome of the experimental study is related to maintaining cardiovascular health after SCI2 (could be interventions specifically aimed to help gain movement or walk, but cardiovascular health is secondary outcome). 4A secondary outcome of the observational study is related to maintaining cardiovascular health after SCI2 (could be interventions specifically aimed to help gain movement or walk, but cardiovascular health is a secondary outcome.). 5A prevalence/incidence study, not fitting in the first 4 categories, where the main focus of the study is to learn about the rate of exercise & overall health in those with SCI.

29. Next Steps • Data Extraction • Create Tables of Evidence

30. Example: Study Flow Chart

31. Grade the Evidence • Most common approaches to classification and grading of the evidence: • American Academy of Neurology (AAN) • GRADE (Grading of Recommendations Assessment, Development and Evaluation) • Adopted by Cochrane Collaboration, WHO, and many others)

32. AAN • Evidence classified class I (low risk of bias) through IV (very high risk of bias) • Evidence graded A (consistent with class 1 studies) and recommend treatment as established as safe and effective • through D (class 4 studies or a mix of 4 classes). Insufficient evidence to support a conclusion of effectiveness

33. GRADE • Classified high, moderate, low and very low • Randomized trial = high • Observational study = low • Any other evidence = very low • Note: in GRADE, look at study design first but there are many other criteria to consider in assigning a grade of evidence • Recommendations (strong or weak) based on classification

34. Steps in Conducting a Systematic Review

35. Moving from article to next step • Hasn’t traditionally been researchers' role • Requires new partnerships • With consumer and advocacy groups • With organizations that create clinical guidelines • NIDRR’s response • Create KT centers such as the MSKTC • Our role is to assist researchers in this new role

36. Systematic Review to User Product Using the synthesis of evidence resulting from you review of the adverse effects of exercise for individuals with SCI • What would you take as a next step for each of the following audiences • CLINICIANS • RESEARCHERS • CONSUMERS

37. Consumer & Clinician Materials:Lessons learned • Provide resources to learn more • Use visuals • For consumers, use and define medical terms • Make it understandable for everyone • Use appropriate tone • Validate findings with target audience • e.g., cognitive interviews Best Practices in Developing Consumer Information: http://msktc.washington.edu/infoneeds/BestPractices_ConsumerInfo.pdf

38. Key Points To Remember Systematic reviews • assemble, critically appraise, and synthesize the results of primary investigations addressing a specific topic or problem • are prepared using strategies that limit bias and random error • can help practitioners keep up to date with the overwhelming volume of medical literature • can help ground clinical decisions in research evidence, although they neither make decisions nor obviate the need for sound, compassionate clinical reasoning • help consumers understand what best evidence is

39. Summary of Lessons Learned • Ask the right clinical question (pragmatic) • When there is not adequate evidence yet… • Get consumer input • Don’t stop when the review is completed

40. For More Information • Dijkers, M. When the Best is the Enemy of the Good: The Nature of Research Used in Systematic Review and Guidelines (http://www.sedl.org/pubs/catalog/items/dis125.html). • Johnston, M., Vanderheiden, Farkas, Rogers, Summers & Westbrook, The Challenge of Evidence in Disability and Rehabilitation Research and Practice (online soon at www.ncddr.org/kt/products/tfpapers/tfse_challenge/) • Sudsawad, P. (2007) Knowledge Translation: Introduction to Models, Strategies and Measures (http://www.sedl.org/pubs/catalog/items/dis112.html) • American Academy of Neurology (AAN) Clinical Practice Guideline Process Manual (http://www.aan.com/globals/axon/assets/3749.pdf) • GRADE (http://www.gradeworkinggroup.org/)