How to plan and manage an environmental education program – AND evaluate its success Philip Cox,

1. How to plan and manage an environmental education program –AND evaluate its successPhilip Cox, Plan:Net Limited (www.plannet.ca) Alberta Council for Environmental Education (ACEE) October 18th Webinar

2. A craft that comes by different names… Outcomes Measurement Performance Measurement and Evaluation Outcomes Mapping Managing by Results Results Based Management

3. A craft that comes by different names… Outcomes Measurement Performance Measurement and Evaluation Outcomes Mapping Managing by Results Results Based Management

4. Introducing Three Outcomes Measurement Tools… Logic Model Risk Analysis and Management Table Monitoring and Evaluation Plan

5. Implementation Project Initiation Proposal Development Donor Appraisal Start Up Monitoring / Evaluation Phase Out The basis for workplans and job descriptions A management reference during team meetings A guide for monitoring and reporting on progress & adjusting strategies An aid in designing an evaluation The centrepiece of your proposal or plan 1. The basis for an introductory conversation with stakeholders Logic Model, Risk Analysis and Management Table, and Monitoring and Evaluation Plan are relevant to all parts of the program cycle…

6. What is Outcomes Measurement? • It is a way of thinking and a set of tools to help us: • Convert a project idea into a set of activities and anticipated changes in a way that is: • Mindful of context • In keeping with the time and resources available • Keep track of progress against the plan and use this information to: • Make management decisions, • Service accountability relationshipsand • To be a learning organization.

7. Outcome Measurement is Part of a Global Trend • A response to new realities... • Need to demonstrate value to citizens and taxpayers… • A response by funding bodies around the globe who are under pressure to provide evidence that their programs achieve results • EEC, World Bank, USAID, UN, WFP and others need to provide credible evidence of results they achieve • By mid 90’s, almost all OECD governments had adopted RBM approaches • In Canada - federal government and all provinces • Increasingly philanthropic organizations such as Alberta Ecotrust Foundation, EcoAction, etc.

8. What does this mean in practical terms? • When making a plan, or reporting on progress, we are challenged like never before to: • Describe the connection between the work we do and the difference we make

9. If you are planning… • A new web-based information portal for educators on climate change • You might focus attention on… • The quantity and quality of information loaded on to the portal • The ‘user-friendliness’ of the web site • The number of users and how they are accessing the portal • And also… • What teachers are looking for on the portal • How they are using the information to inform their teaching • How the teaching-learning experience in the classroom is altered

10. Exercise • Think about a project that you know well • Craft two outcome statements that reflect the change you want to see • Share your statement with your neighbour and then in plenary

11. Where can Outcomes Measurement be used in your organization? • Organization as a whole • Donor funded programs • Partner organizations • Individual Partner Initiatives/Projects

12. Three Outcomes Measurement Tools Logic Model  - “Measuring Success…” pages 19 - 23 Risk Analysis and Management Table Monitoring and Evaluation Plan

13. Outcomes Measurement practice revolves around the logic model The Logic Model

14. Splashand Ripple ! Another way of thinking about Activities and Outcomes Splash and Ripple!

15. Situation Assessment Before beginning a results based plan.. 1. Looking Inward - Mandate • Competencies • Experience 2. Looking Outward • Issues & Opportunities 3. Clarifying Priorities

16. Situation Assessment - Stakeholders Influence HIGH High Interest/ Importance, High Influence High Interest/ Importance, Low Influence Interest or Importance Consider ways to strengthen or protect interests Build coalition of support Low Interest/ Importance, High Influence Low Interest/ Importance, Low Influence LOW Spend more time on the other Stakeholders Caution - be aware of risks or obstacles HIGH LOW

17. Discussion • Generate a list of groups with a stake in the success of your project • Place them on the stakeholder matrix according to their current • Interest/importance • Influence

18. Inputs Activities Outputs Activities Outputs Activities Outputs Outputs Outputs Activities Activities Outcomes Inputs Outcomes Impact Inputs Inputs Outcomes Inputs Logic Model Format - One Version

19. Ultimate Outcomes Activities Outputs Outputs Outputs Outputs Outputs Activities Activities Activities Activities Intermediate Outcomes Intermediate Outcomes Intermediate Outcomes Immediate Outcomes Immediate Outcomes Immediate Outcomes Immediate Outcomes Inputs Logic Model Format - Another Version Why What we want How

20. Inputs… • Are the human and physical resources that make a program or project possible. • Links outcomes logic to budgeting

22. Activities • Describe the essential work of the program or project – what you DO • Should be summarized in no more than 5 statements.

24. Outputs… • Short term …a change that is observable as you complete the activity. • Each output relates to one activity. • You have substantial control. • Involves • a specific group of people - those directly involved in the activity, or • the creation of a product • The change helps make outcomes possible.

26. Immediate Outcomes… • Observable within the project cycle, following activities • Describes the potential created by the outputs • You still have considerable control • Centres on a very specific group of people - those directly involved in the activity or using the product.

28. Intermediate Outcomes… • Describe the key changes you want to make in the program/project • Observable at or shortly after the completion of the program/project • You have influence, but not control • Usually involves a larger group of people than those directly involved in the program/project • Are fewer in number; each flows naturally from several Immediate Outcomes

30. Ultimate Outcome • Long term …a picture of a preferred future; the reason why the program or project is important. • You can only make a contribution and have an indirect influence. • The change usually involves an even wider group of people • Is observable well after the program or project is complete.

32. A Progression of Results - Typical Terms Ultimate Outcomes Conditions - Social Economic Civic Cultural Environmental Intermediate Outcomes Performance Improved - Lifestyle shifts Institutional behaviour changes - policy & programming Social mobilization Immediate Outcomes Capacities Developed - Institutional exchanges/collaborations Applied knowledge Functioning spaces/infrastructure System improvements/adaptations New/Improved leadership Aligned Decision-making Outputs What we Produce - Networks Courses Trained participants Assessments Publications Infrastructure Activities What we Do - Research Build Train/Mentor Promote Procure Convene Inputs What we Need - Staff Equipment Supplies Office Space Services

33. Logic Model Quiz “Organize a Provincial Climate Change Summit”

34. Making a Logic Model - Some Tips • Vantage Point • Logic Models can be written from many different points of view - • it is important to be clear who is doing the activities - spending implementation funds • there is normally only one vantage point in a framework, it could be • a single entity, or • shared across two or more entities that are working in partnership

35. When building an logic model consider three variables… • Time • Outputs - Immediate • Immediate Outcomes - within time-frame following related activities • Intermediate Outcomes - by the end of the project or shortly afterward • Ultimate Outcomes - beyond the project • Control • Outputs - substantial control • Immediate Outcomes - still considerable control • Intermediate Outcomes - direct influence • Ultimate Outcome - indirect influence • Diffusion • Outputs - persons directly involved in the activity • Immediate Outcomes - same, possibly early interaction with additional people closely connected (e.g. users, colleagues, family members) • Intermediate Outcomes - larger teams, organizations, population segments • Ultimate Outcome - systems (communities, networks, societies)

36. Proportion… • Outputs, outcomes and impacts vary in their magnitude according to how big (in money terms) and how long the project is. • If you have a three week, $5,000 project, your outcomes should reflect what is reasonable to expect at three weeks and after $5,000 has been spent • If you have a three year, $500,000 project, your outcomes should reflect what is reasonable to expect at three years and after $500,000 has been spent

37. Scale - macro to micro Outcome Measurement can be used at any scale - but which is best for you?

38. Writing Activity and Results Statements - Tips • Separating the Action from the Change • Activities - where you describe what you, as the implementing group, are DOING • Results (output and outcome - immediate, intermediate and ultimate) - where you describe what is DIFFERENT • For example… • Activity: Train 25 participants in Outcomes Measurement • Result: Participants apply Outcomes Measurement knowledge in when making project plans

39. Avoid ‘run-on’ statements - for example: “Rural population retained in rural areas THROUGH improving their living conditions BY MEANS OF enhancing rural income, better access to health, social services and infrastructure.” • An impact statement for a rural development project “Improved political freedoms of local communities TO access, participate and CONTRIBUTE TOWARDS sustainable regional development.” - An impact statement for a governance related project Difficult to locate the actual change you seek

40. Do not lose sight of the people in your outcomes statements. • Sometimes we write statements without naming the people or groups who are experiencing the change. For example… “increased awareness” • Try to include the subject in the statement at all times

41. Avoid tentative wording like: “able to”,“have increased capacity to…”,“contribute toward”. These are vague in meaning and very hard to measure.

42. Now your turn….. Describe your group project in a results logic model? 1 Ultimate Outcome 2 Intermediate Outcomes 3-4 Immediate Outcomes 4-6 Activities and Outputs

43. Sample Logic Model Worksheet

44. Three Outcomes Measurement Tools Results Logic Model Risk Analysis and Management Table Monitoring and Evaluation Plan

45. Enablers and Constraints… • Are forces affecting any part of your project both positively and negatively

46. Effects Significant (3) Monitor Risks Risk Mitigation Required Risk Mitigation Required Moderate (2) Risk Mitigation Required Monitor Risks Acceptable Risks Minor (1) Monitor Risks Acceptable Risks Acceptable Risks Low (1) Medium (2) High (3) Likelihood Risk Analysis Matrix Project Example… Climate Change Conference Immediate Outcome: Conference participants deliberate across stakeholder lines on conference topics; they identify climate change strategies around which there is common agreement, as well as strategy areas requiring further discussion 1. What could hamper progress? 2. What is the likelihood of a problem? 3. What effect would the problem have on the Outcome?

47. Risk Mitigation & Monitoring • Risk mitigation strategies should reduce the likelihood of an undesirable event, or minimize the effect the event could have on the project. • Risk areas that remain above the level of acceptable risk should be monitored using risk indicators.

48. A Typical Risk Analysis and Management Table

49. Three Outcomes Measurement Tools Logic Model Risk Analysis and Management Table Monitoring and Evaluation Plan - “Measuring Success…” Appendices, pages 44 - 66

50. Monitoring and Evaluation - The Distinction • Monitoring A continuous (or regular) self-assessment of progress -collecting data and comparing current performance with planned activities and outcomes. • Evaluation Independent (often external), periodic, strategically focused assessment of a program/project’s continuing relevance, management, governance, results, coverage, external relationships, sustainability, etc.