Nicola Lauzon, Marc Turcotte & Laura McAuley Impact Assessments Unit

1. Evaluating the impact of health research:Revisiting the Canadian Institutes of Health Research Impact Assessment Framework Nicola Lauzon, Marc Turcotte & Laura McAuley Impact Assessments Unit Canadian Institutes of Health Research Ottawa, Ontario, Canada nicola.lauzon@cihr.gc.ca

2. Presentation Overview • CIHR impact assessment framework history • Study purpose & methods • Findings • Uses of CIHR framework • Focus group feedback from framework users • Application of Lessons learned • Changes to the framework categories • Development of a simplified conceptual model • Next steps • Questions

3. CIHR Impact Assessment Framework 2006

4. Study Purpose & Methods • Purpose • To document lessons learned and inform framework improvement • Methods • Document review of impact reports & protocols • How has it been used, results of use, lessons learned • Focus groups • Feedback and input from users and potential users of framework • Six groups; total 35 participants

5. Results: Document Review • Uses of CIHR framework • Topic-based assessment • Focus on specific disease condition • Investigate impacts across multiple categories • Institute-based assessment • Focus on research mandate areas of CIHR Institutes • Investigate impacts across multiple categories • Category-based assessment • Focus exclusively on one impact category

6. Topic-based impact assessments • Topic • Cardiovascular disease • Obesity • SARS • Main Lessons learned • Validation of CIHR impact categories • Impacts continue after 20+ years • Attribution difficult • Topic dictates types of impacts seen

7. Institute-based impact assessments • CIHR International Review (2010 – 2011) • Impacts of broad mandate areas of all 13 CIHR Institutes between 2000 – 2010 • Lessons learned • Limited availability of indicators in latter impact categories • Need to revisit the categorization and utility of certain indicators • Potential to improve bibliometric methodology

8. Results: Focus Groups • Main theme of discussions: • Attribution versus contribution • Direct attribution desired • Struggle to find value of latter categories and associated indicators • Lessons Learned • Need to clarify to stakeholders: • Our conceptualization of the research impact process • How CIHR fits into the research impact process • How the CIHR framework fits into the research impact process

9. Simplified conceptual model of research impact Society Research Enterprise Research Users Use Effect Findings Funding Research Knowledge Translation Health Knowledge Impacts on… Decision-Making Health System Capacity Economy

10. Not-so-simplified version Society Research Enterprise Research Users Use Effect Findings Funding Research Knowledge Translation Health Knowledge Impacts on… Decision-Making Health System Capacity Economy

11. Not-so-simplified version Society Research Enterprise Research Users Publication bias Research Institution priorities User characteristics Partnerships & leveraged funds Government priorities Researcher priorities Use Effect Funding Findings Research Peer review System capacity Political climate Demand Government priorities Knowledge Translation Funder priorities Public priorities Infrastructure Peer review Economic outlook Government budget Research team composition Accessibility System capacity Health Continued relevance Personal characteristics Researcher characteristics Time lag to publication User acceptance Knowledge Impacts on… Decision-Making Health System Capacity Economy

12. Simplified conceptual model of research impact Society Research Enterprise Research Users Use Effect Findings Funding Research Knowledge Translation Health Knowledge Impacts on… Decision-Making Health System Capacity Economy

13. Simplified conceptual model of research impact Society Research Enterprise Research Users Use Effect Findings Funding Research Knowledge Translation Health Knowledge Impacts on… Decision-Making Health System Capacity Economy

14. Simplified conceptual model of research impact Society Research Enterprise Research Users Use Effect Findings Funding Research Knowledge Translation Health Knowledge Impacts on… Decision-Making Health System Capacity Economy

15. Simplified conceptual model of research impact Society Research Enterprise Research Users Use Effect Findings Funding Research Knowledge Translation Health Knowledge Impacts on… Decision-Making Health System Capacity Economy

16. CIHR Impact Assessment Framework

17. Next steps & Conclusions • Continue testing model, framework categories and indicators • Feedback & validation from stakeholders • Alignment of model and framework with literature

18. Questions???