Community Partnership Evaluation for Injury Prevention

1. Community Partnership Evaluation for Injury Prevention Susan J. Snelling, PhD Health Promotion Field Support Specialist, Evaluation

2. Instructions • Audio: You can listen and speak through your computer or phone – this is an option for you to select in the “Audio Mode”

3. Instructions • Question: When it comes to the Q & A session you can let me know you have a question by clicking the “Raise Your Hand” button

4. Instructions • Question: If you have a question you would like to type, you can do so in the textbox provided

5. WEBINAR OUTLINE • Partnerships • What is a partnership? • Why do we work in partnerships? What are the challenges? • How do partnerships work and what makes a successful partnership? • Evaluating Community Partnerships • What is evaluation? • Options for evaluating partnerships: Strategies, evaluation options, measures, methods, and tools • Resources

6. What is a Partnership?

7. Different Types of Inter-Organizational Work • Networking: The exchange of information for mutual benefit. • Coordination: Exchanging information and altering activities for a common purpose. Match and coordinate needs and activities. Limit duplication of services. • Cooperation: As above plus sharing resources. • Collaboration: As above plus, collaboration includes enhancing the capacity of the other partners for mutual benefit and a common purpose. Longer-term and more deliberate efforts of organizations and groups to undertake new, joint activities. Building interdependent systems to address issues and opportunities. Adapted from Labonte (2003) and Integrated Health Promotion(2003), Victoria, Australia.

8. What is a Partnership? Community Partnership involves building collaborative relationships between different groups in the community to achieve a common goal of promoting health. Within the partnership, decision-making is shared across partners. The mutual action of the partnership has the potential to achieve more than any one partner working independently. - (Sears 2009 for Heart Health Resource Centres)

9. Whydo we work in Partnerships in injury prevention?

10. Why do we Work in Partnerships? • Resource limitations; share workload • Reduce duplication • Increase influence and ‘reach’ • Include a variety of perspectives • Learn from others • Develop shared messaging; reduce public confusion • Ontario Public Health Standards are built on a principle of partnership and collaboration • Social change goes beyond public health: see OPHS logic models!

11. What Challenges do you experience in injury prevention partnerships?

12. What Challenges do we Experience in Partnerships? • Processes: How we work together • Partnership building and maintenance takes time • Difficulty coming to consensus may impede current and future efforts • Power imbalances; contribution inequities • Changes in membership • Outcomes: What we achieve together • “Opportunity costs” of engaging in partnerships • Focus on information sharing or endless planning rather than achieving results • Focus on long-term goals can lead to loss of focus and commitment in the short term Adapted from Partnerships: The Benefits. (2007) Institute of Public Health in Ireland.

13. How do partnerships work and what makes a successful partnership?

14. Three Levels at Which Partnerships Operate • Level 1: Partnership infrastructure, function, and processes • Level 2: Partnership programs and interventions • Level 3: Health and systems change outcomes

15. Five Conditions of Collective Success • Common agenda • Shared measurement systems • Mutually reinforcing activities • Continuous communications • Backbone support organizations - John Kaniaand Mark Kramer, “Collective Impact” (2011)

16. what is evaluation?

17. What is Evaluation? The application of systematic methods to address questions about program operations and results. • Wholey, 2010, Handbook of Practical Program Evaluation

18. 10 Steps in Evaluation Process Step 1: Clarify your program Step 2: Engage stakeholders Step 3: Assess resources for evaluation Step 4: Select the type of evaluation Step 5: Determine methods of measurement and procedures Step 6: Develop workplan, budget and timeline Step 7: Collect data Step 8: Process the data and analyze the results Step 9: Interpret and disseminate results Step 10: Take action

19. Process Evaluation • How do we do our work? • Measures activity and implementation • Sometimes known as tracking or monitoring

20. Outcome Evaluation • Are we achieving the objectives we set out? • Estimates success of program (or partnership) at meeting goals and objectives • Measures effects of activities at producing change • It can be difficult to measure long-term outcomes • Ii can be difficult (or impossible) to isolate the contribution of your work to a societal result

21. Options for evaluating partnerships

22. Strategies for Implementing Effective Evaluation in Partnerships • Build in evaluation concepts from the start of the partnership • Establish principles: Here are a few suggestions: • No shame, no blame – do evaluation to learn and improve • Don’t evaluate something that can’t change • Commit to taking action based on the results • ‘We’re all in this together’ (the problems and the solutions are not the leader’s responsibility) • Establish clear evaluation questions • Have a plan for using the results of an evaluation

23. Options for Evaluating Partnerships • Start with Evaluability Assessment see Butterfoss: Are You Ready to Evaluate Your Coalition? http://coalitionswork.com/documents/are_you_ready_to_evaluate_your_coalition.pdf • Since there are three levels at which partnerships operate, there are also three levels on which evaluation can focus: • Level 1: Partnership infrastructure, function, and processes • Level 2: Partnership programs and interventions • Level 3: Health and systems change outcomes

24. Measures for Evaluating Levels of Partnerships Level 1: Structures and Functions • Number and type of partners • Skills of partners • Diversity • Commitment, sense of ownership • Partnership processes such as meetings, workload • Leadership style and effectiveness • Member satisfaction • Perceived effectiveness among members and community Level 2: Program Implementation • Implementation of plan • Programs and services offered • Community actions taken • Reach; Media coverage • Knowledge and behaviour change • Improved or new policies or practices (Level 2 or 3) Level 3: Outcomes and Impacts • Health status • Community changes • Community capacity • Partnership sustainability

25. Methods and Tools for Evaluating Partnerships Methods • Active data collection: Surveys, interviews, focus groups, self-report • Unobtrusive data collection: Observation, monitoring, statistics • Descriptions of partnerships can be done through network mapping Tools • Coalition Member Survey; Coalition Effectiveness Inventory (Butterfoss) • Vic Health Partnerships Analysis Tool • Network Health Scorecard • Mental Health Mapping Tool • Partnership Effectiveness Tool

26. RESOURCES

27. Health Promotion Capacity Building Services • Free to those working on Ontario-focused projects. • Scope varies, depending on need: • training sessions; • brief, one-time advice; • consultations; • review your work or product; • links to other sources of information and resources. • Consultation request form http://www.thcu.ca/consultation/request_form.cfm

28. Questions?

29. Thanks to Ontario Injury Prevention Resource Centre for supporting us on this webinar!

30. Public Health Ontario wishes to acknowledge and thank THCU staff and many partners who contributed to an earlier version of this document. THCU (originally known as The Health Communication Unit, started in 1993 at the University of Toronto) moved to Public Health Ontario’s Health Promotion, Chronic Disease and Injury Prevention Department in 2011.