Improving the Quality in Environmental Evaluations

1. Improving the Quality in Environmental Evaluations 2010 Canadian Forum of the Environmental Evaluators Network Presenter: Gavin Lemieux Evaluation Division October 1st, 2010

2. Content • Context • Improving the Utility of Evaluations • Improving the Quality of our Evaluation Process • Discussion

3. Environment Canada • Environment Canada's mandate is to preserve and enhance the quality of the natural environment; conserve Canada's renewable resources; conserve and protect Canada's water resources; forecast weather and environmental change; enforce rules relating to boundary waters; and coordinate environmental policies and programs for the federal government. • The department employs about 6000 people and has an annual budget of close to a billion dollars. Approximately 60 percent of its workforce and 80 percent of its budget is devoted to science and technology activities. • Environment Canada is required to maintain an independent evaluation capacity for the purpose of providing objective and evidence-based information and advice on the performance of programs to enhance results-oriented and accountable management. • Environment Canada’s Evaluation Division is comprised of one director and eleven evaluators. All our approved reports are available on our website: www.ec.gc.ca/ae-ve.

4. Evaluation Context Within the Evaluation Division at Environment Canada, quality is an important value to ensure departmental senior management have credible information on the effectiveness of departmental programs. • focus on continuous improvement of our practice to increase the overall effectiveness of the evaluation function. • various tools and procedures have been implemented with the overall goal of improving the utility and quality of our evaluation practice. • OAG highlighted several of these tools in recent audit of evaluation functions.

5. Improving the Utility of Evaluations

6. Evaluations Undertaken since 2006-07 Types of evaluation recommendations/lessons learned • Results of a content analysis of recommendations/ lessons learned from approved EC evaluations from 2006-07 to now (29 evaluations, 158 recommendations/lessons learned) - 30% - program design - 21% - communication/consultation/coordination - 20% - financial/performance measurement & reporting - 15% - program delivery/capacity/resources - 8% - governance structures & strategies - 7% - roles & responsibilities

7. Lessons Learned • After each evaluation, lessons learned are developed and shared with a wider community of managers and personnel involved with related or similar programs or issues. • Learnings help evaluatee improve their program, but are also leading to improvements in other similar/related programs as well across the Department.

8. Recommendations Follow-up Process • Environment Canada regularly monitors and reports on the status of management commitments made in response to evaluation recommendations. • Intent is to provide Environment Canada’s Departmental Evaluation Committee (chaired by the Deputy Minister) with timely information on how well the department is addressing and resolving risks or deficiencies and acting on identified opportunities. • Follow-up process ensures recommendations and management response and action plan, approved by DEC, are fully implemented, and that senior management is kept apprised of progress in addressing recommendations/action plan implemention.

9. Departmental Challenge Function • The Evaluation Division also provides a challenge function with respect to: • Memoranda to Cabinet; Treasury Board Submissions • Performance Measurement (e.g., Performance Measurement Strategies, Performance Monitoring and Evaluation Plans) • Evaluation is now an integral aspect of the departmental challenge function and our advice is sought after by program managers

10. Discussion • Benefits: • Ensuring that departmental management also receives timely, strategically focused, objective and evidence-based information on the policies, programs, and initiatives of the department. • Ensuring that the evaluation function remains relevant and is perceived as an integral part of the departmental oversight function. • Challenges: • Difficult at times to be able to truly assess program relevance and performance due to a lack of reliable performance and financial information.

11. Improving the Quality of our Evaluation Process

12. 1. Quality Assurance Process • A self-assessment performed annually by a senior evaluator at the end of each fiscal year • Adapted for evaluation from standards developed by the Institute of Internal Auditors • Incorporates Treasury Board Secretariat evaluation standards • Evaluation categories: • Purpose, Authority, & Responsibility; Objectivity & Integrity; Competency; Evaluation Planning & Issues; Measurement & Analysis; and Reporting • Ratings: • Compliant, Partially Compliant, and Non-Compliant • Required actions are also generated through this process • Benefits: increases strategic awareness and focus on standards, reduces time before improvements are implemented

13. 2. Client Surveys • After each evaluation, Client Surveys are sent to assess evaluation products and services • Evaluation Committee members asked about the quality of the evaluation, including process, and the value of the evaluation • Functional ADM asked about the value of the evaluation • Results are documented and communicated to the project team, Director, and Director General • Evaluation Division identifies best practices and improvements • Benefits: provides program and senior management perspective on process and value

14. 3. Time Recording System • Allows measurement of person-time spent on each project by main categories • Effective tool for planning evaluation resource allocation • Enhances ability to establish commitments and deliver on them accordingly • All staff use these password-protected spreadsheets to log their daily activities Benefits: increases level of precision in planning, reduces the risk of ‘surprise’ time-loss

15. 4. Best Practices: Tools & Procedures (cont’d) • Shared drive • Templates

16. Discussion • The OAG audit stressed the importance of continuing to improve our evaluation processes • Benefits: • Greater ability to identify and act on any areas in need of improvement • Improved performance measurement for our function • Challenges: • Learning and implementation curve • Need to monitor the utility of these processes • Difficulty of increasing workload with staffing constraints