Evaluation Presentation to the Brazilian Conference November 23, 2004. Overview. Agriculture in Canada Department of Agriculture and Agri-Food Evaluation Mandate and Profile Evaluation Planning Community Pastures Program Program Overview Evaluation Framework Evaluation Findings
Evaluation Presentation to the Brazilian ConferenceNovember 23, 2004
Agriculture in Canada
Evaluation Mandate and Profile
Audit and Evaluation Mandate
Provide the Departmental Audit and Evaluation Committee with independent assurance, information and advice on the effectiveness, efficiency and economy of programs, policies and operations
Audit and Evaluation
Planning and Coordination
Grants and Contributions
Central Agency Coordination
Advice and Assessements
Planning and Development
Excellence Through Assurance and Information
Key Result Commitments
Assuranceon the soundness of risk management strategies and practices, the design and operation of management control frameworks, systems and practices, and information used for decision making and reporting;
Informationon the performance of departmental policies, programs and initiatives which is timely, strategically focussed, objective and evidence-based.
While the risk-based planning and reporting of our audits, evaluations and assessments is approved by DAET, we are accountable to the DM for managing our function in accordance with the TB policies.
Collaboration for Excellence
We view and treat our work as an integral part of the departmental management process. Our approach is therefore grounded in a collaborative service-oriented philosophy. Departmental managers can help us help them succeed by supplying appropriate, accurate and timely information and ensuring we understand their needs and expectations.
Services are provided in both official languages.
In order to ensure we are able to deliver our mandate, our budget is focussed on projects approved by DAET. Requests not approved by DAET are performed if time permits and on a cost recovery basis.
Our Service Guarantee
We guarantee our work by paying or reimbursing for the hiring of a consulting firm to complete work which does not meet the criteria outlined in the approved Terms of Reference.
Accessibility of our Reports
Once completed and approved, reports, management responses and/or action plans are available on our Web site http://www.agr.gc.ca/review/rb-ep_e.php?page=list98and accessible by the public in accordance with the Government Audit and Evaluation Policies and the Access to Information and Privacy Acts.
Questions or Comments
Our primary point of contact is Frank Brunetta, Director General, Audit and Evaluation Team at (613) 759-6471, by facsimile at (613) 759-6499, or by E-Mail: brunettaf.agr.gc.ca
Audit and Evaluation Budget 2003-2004
Total Budget: $4,700,000
Full-time Equivalents: 27
Note:Indirect Costs includes items such as salaries associated with management and administration of functions, equipment, materials, and supplies.
Preliminary Project Identification
Against Risk Factors
Consult Senior Management,
BoDs, Legal and the OAG
(validate issues, priorities and projects)
Audit & Evaluation Plan
(costing, HR capacity, scheduling)
Table Plan for Approval – April 14
The Risk-Based Strategically Focused
Central Agency Priorities
AAFC Strategic Outcomes
Achievement of Results (including secondary impacts)
The degree to which results have been achieved as well as any secondary effects observed (intended or unintended) that fall out of the iniatives.
Relevance, Appropriateness and Acceptance
Whether there is a continuing need, as well as the extent it meets the needs of all its stakeholders/partners. whether or not the organizational design (delivery mechanisms), terms and conditions and resources are sufficient to meet the objectives and its intended targets.
The comparison of costs (direct and indirect) to the benefits (tangible and intangible) to illustrate the relationships among costs, inputs and outputs and outcomes (benefits).
Insuring management commitment to manage, plan, organize, deliver, monitor, make adjustments, communicate and report on performance is clearly defined, understood and consistently implemented through appropriate policies, processes, practices and systems
* adapted from the Canadian Comprehensive Audit Foundation (CCAF) Governance Checklist
Achievement of Results over a period of time (including secondary impacts)
Q:In what manner and to what extent were appropriate results achieved due to the program?
Q:What client benefits and broader outcomes, both intended and unintended, resulted from carrying out the program?
Q:In what manner and to what extent does the program complement, duplicate, overlap or work at cross purposes with other programs or organizations within this sector?
Relevance, Appropriateness and Acceptance
Q:To what extent are the objectives and mandate of the program still relevant to the future direction and needs of the sector and consistent with government priorities?
Q:Are activities and the operational work plan consistent with the program mandate and plausibly linked to the stated results and other intended results?
Relevance, Appropriateness and Acceptance (cont.)
Q:Are there more cost-effective ways of delivering the existing program to achieve the stated objectives? What is the usefulness of the current delivery mechanism (are there any alternatives)?
Q:Are clients and other stakeholders satisfied with the program’s activities? Are there formal and informal mechanisms to provide feedback?
Q:What are the costs associated with the program and its various project components, e.g. key activities and their results (outputs, direct impacts)?
Q:What are the benefits to society, the province, the sector, participants and governments?
Q:Are the costs in line with the benefits? How do they compare to the expected vs actual results?
Q:Are funds being distributed at the lowest possible cost?
Typical approaches include:
Minimize data bias by ensuring that for each evaluation issue, there are several independent lines of evidence related to that issue
Community Pastures Program
Original Program Objectives
Current Program Objectives
Sample of Existing Matrix
Direct and Indirect Benefits/Findings
Stated Program Objective #1:
To make possible the removal of lands and to facilitate improved land
Stated Program Objective #2:
To utilize the resource primarily for the summer grazing
World class expertise
CPP is achieving conservation & maintenance goals on existing pastures
Program land base has not grown since 1970's
Role of Department and federal government
Insufficient program information
Focus on original mandate
Look for «Program Mandate Creep»
Lack of program logic model and associated measurement
framework minimizes ability to objectively evaluate
Recognize political realities