developing evaluation capacity n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Developing Evaluation Capacity PowerPoint Presentation
Download Presentation
Developing Evaluation Capacity

Loading in 2 Seconds...

play fullscreen
1 / 16

Developing Evaluation Capacity - PowerPoint PPT Presentation


  • 134 Views
  • Uploaded on

Developing Evaluation Capacity. EEN [Canada] Forum Shelley Borys Director, Evaluation September 30, 2010. Context. The importance of building evaluation capacity is linked to the following factors: Increased requirements for evaluation

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Developing Evaluation Capacity' - alban


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
developing evaluation capacity

Developing Evaluation Capacity

EEN [Canada] Forum

Shelley Borys

Director, Evaluation

September 30, 2010

context
Context
  • The importance of building evaluation capacity is linked to the following factors:
    • Increased requirements for evaluation
      • The requirements under the new Policy on Evaluation to move towards coverage of 100% of both transfer program and direct program spending on a five-year cycle.
    • The increased interest in the use of evaluation evidence to provide useful feedback regarding program improvement, to demonstrate progress toward outcomes, and to support decision-making.
overview
Overview
  • This presentation will discuss ways in which Environment Canada is working to increase its own evaluation capacity.
  • This topic will be explored from three perspectives:
    • How we build evaluation capacity within the Evaluation Division;
    • Activities to increase the capacity for programs to be evaluated; and
    • How we build capacity to use evaluation.
evaluation competencies
Evaluation Competencies
  • Two frameworks exist which elaborate a series of evaluator competencies which can serve as a guide to capacity development:
    • Canadian Evaluation Society Credentialing Initiative
      • Five competency domains: reflective (7), technical (16), situational (9), management (7), interpersonal (10)
    • Treasury Board Secretariat Centre for Excellence in Evaluation
      • Four competency clusters: intellectual (3), management (2), relationship (3), personal (2)
building evaluator capacity environment canada evaluation model
Building Evaluator Capacity:Environment Canada Evaluation Model
  • Intent is to conduct as much evaluation work as is possible in-house
    • Use contractors for evaluations that are technically beyond our capacity, interdepartmental evaluations, and where we have contract dollars but not staff time
  • This helps build evaluation capacity by:
    • Having evaluators gain hands-on evaluation experience
    • Helping evaluators become subject-matter experts
    • Facilitating evaluators to develop a network of contacts within the Department
building evaluator capacity project teams
Building Evaluator Capacity:Project Teams
  • Evaluation Division teams are formed for each project, with:
    • A project lead, who coordinates communications, directs the work of team members, and is the lead pen on deliverables;
    • Team members, who support the efforts of the project lead in performing evaluation activities and preparing reports.
    • A peer reviewer, who is another evaluation manager whose role is to review in-progress deliverables, before they are submitted for management review; and
  • Teams are not necessarily hierarchical.
  • This helps build evaluation capacity by
      • Providing exposure to different styles of evaluation and project management
      • Providing an opportunity to work on different tasks and a range of projects
      • Enabling evaluators to take on additional responsibilities while managing risk.
building evaluator capacity templates
Building Evaluator Capacity:Templates
  • Evaluation projects use a common file folder structure and basic document formats.
  • These templates build evaluation capacity by:
    • Ensuring that all required issues are addressed;
    • Laying out a somewhat standard process for evaluation;
    • Reducing the risk of uneven evaluation conduct.
building evaluator capacity generic evaluation framework
Building Evaluator Capacity:Generic Evaluation Framework
  • One templated item of particular importance is the generic evaluation framework.
  • This is a matrix table contained within the evaluation plan template that sets out:
    • Key questions to be asked;
    • Indicators for each question;
    • Sources and methods that would support the analysis; and
    • The specific Treasury Board Secretariat Policy on Evaluation issue each question relates to.
  • This evaluation framework helps build evaluator capacity by:
    • Ensuring that all evaluations begin from the same foundation
    • Serving as guidance around the identification of indicators and data sources.
building evaluator capacity training
Building Evaluator Capacity:Training
  • Staff within the Evaluation Division are encouraged to seek out opportunities for training and professional development
    • Conferences, networking events, lunch and learn sessions
    • Resource library
  • This helps to build evaluation capacity by:
    • Helping evaluation staff remain abreast of developments within the theory and practice of evaluation
building capacity to be evaluated shadow plan
Building Capacity to be Evaluated: Shadow Plan
  • Two years in advance of a planned evaluation, we conduct a mini evaluability assessment:
    • Performance measurement?
    • Logic Model?
    • Key issues of interest?
    • Timing consideration?
  • Intent is that improvements can be introduced before the evaluation starts, thereby strengthening the evaluation
    • Performance measurement
    • Relevance to managers
building capacity to be evaluated performance measurement improvements
Building Capacity to be Evaluated:Performance Measurement Improvements
  • The Evaluation Division regularly assesses the quality of performance measurement in its evaluations, as shown in the following table:
  • In recommending improvements to performance measurement, subsequent evaluations will have better data to draw upon.
building capacity to be evaluated and to use evaluation integration with program design
Building Capacity to be Evaluated and to Use Evaluation: Integration with Program Design
  • The Department has established several mechanisms for the Division to provide feedback on potential programs at an early stage. The Division is consulted on:
    • Memoranda to Cabinet submissions;
    • Treasury Board Secretariat submissions; and
    • The design of performance measurement strategies.
  • This helps build the capacity to be evaluated and to use evaluation by:
    • Establishing a dialogue between program staff and evaluators in advance of evaluation;
    • Ensuring evaluation results are considered in the design of new initiatives; and
    • Ensuring that program design supports evaluation e.g. defined outcomes, performance measurement.
building capacity to use evaluation management response guidelines
Building Capacity to Use Evaluation:Management Response Guidelines
  • The Evaluation Division has prepared guidelines for management to draw upon when preparing their response to evaluation recommendations.
  • These guidelines describe the essential components of a management response:
    • In particular, the development of specific actions with deadlines and clear responsibilities.
  • This helps build the capacity to use evaluation by:
    • Ensuring the clarity of management responses; and
    • Facilitating the measurement of progress against action plans.
building capacity to use evaluation follow up process
Building Capacity to Use Evaluation:Follow-up Process
  • After evaluation reports and the management responses to their recommendations have been approved, the Department works to ensure that these recommendations are followed through on.
  • This is done formally at the Departmental Evaluation Committee meetings, where the status of actions stemming from past evaluations is a standard agenda item.
  • This helps build the capacity to use evaluation by:
    • Formalizing the implementation process;
    • Ensuring that senior management remain aware of outstanding action items; and
    • Providing a forum for any barriers to implementation to be discussed and responded to.
building capacity to use evaluation management support
Building Capacity to Use Evaluation:Management Support
  • A successful evaluation unit requires a suitably enabling environment.
  • At Environment Canada, senior management understanding and support of evaluation is evident.
  • Departmental Evaluation Committee is active and engaged
  • This helps build evaluation capacity by:
    • Signalling to program personnel that evaluation is seen as a furtherance of departmental work – not an interruption; and
    • Strengthening the connection between evaluation evidence and senior decision-making.
summary of practices
Evaluator Capacity

In-house Conduct

Team Structure

Templates

Generic Evaluation Framework

Training

Capacity to be Evaluated / Use Evaluation

Performance Measurement Improvements

Shadow Plan

Integration with Program Design

Management Response Guidelines

Follow-up Process

Management Support

Summary of Practices