Evaluation presentation to the brazilian conference november 23 2004
1 / 34

Evaluation Presentation to the Brazilian Conference November 23, 2004 - PowerPoint PPT Presentation

  • Uploaded on

Evaluation Presentation to the Brazilian Conference November 23, 2004. Overview. Agriculture in Canada Department of Agriculture and Agri-Food Evaluation Mandate and Profile Evaluation Planning Community Pastures Program Program Overview Evaluation Framework Evaluation Findings

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Evaluation Presentation to the Brazilian Conference November 23, 2004' - nijole

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Evaluation presentation to the brazilian conference november 23 2004

Evaluation Presentation to the Brazilian ConferenceNovember 23, 2004


  • Agriculture in Canada

  • Department of Agriculture and Agri-Food

  • Evaluation Mandate and Profile

    • Evaluation Planning

      • Community Pastures Program

        • Program Overview

        • Evaluation Framework

        • Evaluation Findings

        • Lessons Learned

  • Questions

Agriculture in canada
Agriculture in Canada

  • Accounts for some $112 billion in annual retail and food service sales

  • Is $81 billion of total economy

  • Represents about 8% of the country's total Gross Domestic Product

  • Is world's third-largest agri-food exporter behind the United States and the European Union

  • Contributes some $7 billion to Canada's annual trade balance, represents about 10 % of the total Canadian trade surplus

  • Provides 1 of every 8 jobs

Agriculture and agri food 2002 2003 expenditures
Agriculture and Agri-Food 2002-2003 Expenditures

Audit and Evaluation Mandate

Provide the Departmental Audit and Evaluation Committee with independent assurance, information and advice on the effectiveness, efficiency and economy of programs, policies and operations

Audit and evaluation organization
Audit and EvaluationOrganization

Frank Brunetta

Director General

Audit and Evaluation

Brigitte Marois


Financial Audit

Richard Hill


Planning and Coordination

Rosemary Stephenson



Eric Seraphim



Grants and Contributions



Special Investigations

Central Agency Coordination

Advice and Assessements


Planning and Development

Information Management


Special Investigations


Audit and evaluation profile
Audit and Evaluation Profile


Excellence Through Assurance and Information

Key Result Commitments

Provide DAET:

Assuranceon the soundness of risk management strategies and practices, the design and operation of management control frameworks, systems and practices, and information used for decision making and reporting;

Informationon the performance of departmental policies, programs and initiatives which is timely, strategically focussed, objective and evidence-based.

  • Our Services

  • Our key Result Commitments are delivered through a comprehensive range of services:

  • planning and conduct of independent audits, evaluations and other reviews approved by the DAET and/or Treasury Board

  • follow-up on management action plans developed in response to audit and evaluation recommendations;

  • advice and consultation on various departmental priority areas;

  • advice on the design of manager-led reviews;

  • management consulting and problem solving support;

  • coordination of Central Agencies activities within the department; and

  • special examinations as directed by the Deputy Minister (DM)


While the risk-based planning and reporting of our audits, evaluations and assessments is approved by DAET, we are accountable to the DM for managing our function in accordance with the TB policies.

Collaboration for Excellence

We view and treat our work as an integral part of the departmental management process. Our approach is therefore grounded in a collaborative service-oriented philosophy. Departmental managers can help us help them succeed by supplying appropriate, accurate and timely information and ensuring we understand their needs and expectations.

Audit and evaluation profile cont
Audit and Evaluation Profile (cont.)

  • Our Service Standards

  • Our work is conducted in a planned, systematic manner while adhering to established standards.

  • We measure our service performance on four factors: Quality, Timeliness, Value and Professionalism.

  • We ensure that staff or contract personnel undertaking reviews are trained and conversant with the standards and codes of ethics appropriate for the work being done and that our work is conducted in compliance with TB as well as other applicable industry standards:

  • The Institute of Internal Auditors (IIA)

  • Canadian Evaluation Society (CES)

  • Information Systems Audit and Control Association (ISACA) Client Survey


Services are provided in both official languages.

Our Costs

In order to ensure we are able to deliver our mandate, our budget is focussed on projects approved by DAET. Requests not approved by DAET are performed if time permits and on a cost recovery basis.

Our Service Guarantee

We guarantee our work by paying or reimbursing for the hiring of a consulting firm to complete work which does not meet the criteria outlined in the approved Terms of Reference.

Accessibility of our Reports

Once completed and approved, reports, management responses and/or action plans are available on our Web site http://www.agr.gc.ca/review/rb-ep_e.php?page=list98and accessible by the public in accordance with the Government Audit and Evaluation Policies and the Access to Information and Privacy Acts.

Questions or Comments

Our primary point of contact is Frank Brunetta, Director General, Audit and Evaluation Team at (613) 759-6471, by facsimile at (613) 759-6499, or by E-Mail: brunettaf.agr.gc.ca

Audit and Evaluation Budget 2003-2004

Total Budget: $4,700,000

Full-time Equivalents: 27









Note: Indirect Costs includes items such as salaries associated with management and administration of functions, equipment, materials, and supplies.

Preliminary Project Identification

Assess Projects

Against Risk Factors

Consult Senior Management,

BoDs, Legal and the OAG

(validate issues, priorities and projects)

Prepare Recommended

Audit & Evaluation Plan

Resource Allocation

(costing, HR capacity, scheduling)


Table Plan for Approval – April 14

The Risk-Based Strategically Focused

Planning Process

Environmental Scan

Government Priorities

Central Agency Priorities

AAFC Strategic Outcomes

Corporate Risks

  • Materiality

  • Federal/AAFC priorities

  • Sensitivity

  • Complexity

  • Management concern

  • Recent change

  • Results of prior work

  • Time since last audit/evaluation

  • Significance to strategic objectives

  • Interdependencies

Risk Assessment

DM Priorities

Evaluation framework
Evaluation Framework

Achievement of Results (including secondary impacts)

The degree to which results have been achieved as well as any secondary effects observed (intended or unintended) that fall out of the iniatives.

Relevance, Appropriateness and Acceptance

Whether there is a continuing need, as well as the extent it meets the needs of all its stakeholders/partners. whether or not the organizational design (delivery mechanisms), terms and conditions and resources are sufficient to meet the objectives and its intended targets.


The comparison of costs (direct and indirect) to the benefits (tangible and intangible) to illustrate the relationships among costs, inputs and outputs and outcomes (benefits).

Management Effectiveness

Insuring management commitment to manage, plan, organize, deliver, monitor, make adjustments, communicate and report on performance is clearly defined, understood and consistently implemented through appropriate policies, processes, practices and systems

* adapted from the Canadian Comprehensive Audit Foundation (CCAF) Governance Checklist

Associated framework questions
Associated Framework Questions

Achievement of Results over a period of time (including secondary impacts)

Q: In what manner and to what extent were appropriate results achieved due to the program?

Q: What client benefits and broader outcomes, both intended and unintended, resulted from carrying out the program?

Q: In what manner and to what extent does the program complement, duplicate, overlap or work at cross purposes with other programs or organizations within this sector?

Relevance, Appropriateness and Acceptance

Q: To what extent are the objectives and mandate of the program still relevant to the future direction and needs of the sector and consistent with government priorities?

Q: Are activities and the operational work plan consistent with the program mandate and plausibly linked to the stated results and other intended results?

Associated framework questions1
Associated Framework Questions

Relevance, Appropriateness and Acceptance (cont.)

Q: Are there more cost-effective ways of delivering the existing program to achieve the stated objectives? What is the usefulness of the current delivery mechanism (are there any alternatives)?

Q: Are clients and other stakeholders satisfied with the program’s activities? Are there formal and informal mechanisms to provide feedback?


Q: What are the costs associated with the program and its various project components, e.g. key activities and their results (outputs, direct impacts)?

Q: What are the benefits to society, the province, the sector, participants and governments?

Q: Are the costs in line with the benefits? How do they compare to the expected vs actual results?

Q: Are funds being distributed at the lowest possible cost?

Program evaluation methodology
Program Evaluation Methodology

Typical approaches include:

  • Review of monitoring data from logic model or performance framework

  • Examination of reports and documentation

  • Statistical analysis

  • Benchmark analysis

  • Economic impact (quantitative)

  • Interviewing stakeholders

  • Etc.

Program evaluation methodology1
Program Evaluation Methodology

Minimize data bias by ensuring that for each evaluation issue, there are several independent lines of evidence related to that issue

Community pastures program overview
Community Pastures Program Overview

  • Land and homestead granting responsibility transferred from federal to provincial in 1930

  • Significant amount of fragile land was homesteaded and cropped

  • Prolonged drought of the 1930s resulted in serious soil erosion problems

  • Resulted in passage of the Prairie Farm Rehabilitation Act in 1935

  • Act: “ ...secure the rehabilitation of the drought and soil drifting areas...and to develop and promote within those areas, systems of farm practice, tree culture, water supply, land utilization and land settlement that will afford greater economic security...through the development, construction, promotion, operation and maintenance of any project or scheme...or agreements with any province, municipality or person...”.

Community pastures program overview1
Community Pastures Program Overview

  • Program established in cooperation with Saskatchewan and Manitoba. Alberta developed own program

  • Program governed by a series of Federal-Provincial agreements with Saskatchewan (1938, 1949, 1964, 1966, 1978) and Manitoba (1939, 1964, 1978)

  • Agreements establish legal authority and define terms and conditions:

    • Provinces acquire and place lands under federal control

    • Canada responsible for the development, operation and maintenance

  • Land base involves 2.3 million acres of erosion-prone marginal crown land

Original Program Objectives

  • “Land Utilization Program” established in 1937

  • Permanent withdrawal of submarginal prairie land from cultivation

  • Development of such areas for grazing purposes; and

  • Resettlement on suitable farm land of farmers removed from submarginal areas

Current Program Objectives

  • “Community Pastures Program” redefined in 1978

  • To make possible the removal of lands from unsuitable or unacceptable land use and to facilitate improved land use through their rehabilitation, conservation and management; and

  • To utilize the resource primarily for the summer grazing of cattle while assisting in stabilizing small farms and providing breeding bulls to encourage high quality, long term cattle production

Community pastures evaluation framework
Community Pastures Evaluation Framework

Sample of Existing Matrix

Review Framework

Evaluation findings
Evaluation Findings

  • 85 pastures, 24 in Manitoba; 60 in Saskatchewan; and 1 in Alberta

  • 80% of pastures surveyed (during1982-98) were in good to excellent condition (target of “at least 75%”)

  • Another approximately 3.2 million (ha) class 4,5 & 6 currently provincial jurisdiction and in production

  • Grazing service to approximately 225,000 head belonging to 3,500 patrons

  • 225,000 head represents 6 - 7% of cow-calf sector in Saskatchewan and Manitoba

  • 99-00 approximately $15 million direct costs to AAFC (185 FTEs)

  • Of $15M, $12 million cost-recovered (net cost of $3 million)


Stated Program Objective #1:

To make possible the removal of lands and to facilitate improved land


  • No significant increase of land base since the 1970's

  • Additional 3.2 million (ha) of class 4,5,6 marginal land under provincial jurisdiction and in crop production

    Stated Program Objective #2:

    To utilize the resource primarily for the summer grazing

  • 3,500 patrons utilize grazing service

    • Not targeted ?

    • Impact to income ?

    • 6-7% of cow calf sector in Saskatchewan & Manitoba ?

    • Patron turnover ratio ?


World class expertise

  • Impressive and critical body of knowledge and expertise in land management, rehabilitation and sustainable practices

    CPP is achieving conservation & maintenance goals on existing pastures

    Program land base has not grown since 1970's

  • Despite the emphasis on the removal of land from unsuitable land use in this objective, there has been no significant removal of land from unsuitable use or any significant increase in the land base managed by CPP since the mid 1970's

    Role of Department and federal government

  • The primary focus of the program is pasture and grazing management, there is a need for a policy debate to determine if the Department should be directly involved in the day-to-day operations of community pastures

    Insufficient program information

  • There is insufficient performance information to accurately assess farm income and patron net worth impacts


  • A policy debate be initiated to determine the appropriateness of the Department managing pastures and grazing services to achieve environmental and farm income goals or whether more effective means are available

  • Explore ways to further apply CPP knowledge and expertise to the 3 mil ha of marginal land which continues to be cropped, as well as on the 10 mil ha of rangeland in less than good condition

  • Develop performance indicators to measure progress toward converting marginal land to improved land use, as well as to provide assurance that progress is being made toward farm income goals

Lessons learned from community pastures evaluation
Lessons Learned from Community Pastures Evaluation

Focus on original mandate

  • What was program established to accomplish?

  • What were the intended results?

  • Are they still relevant?

    Look for «Program Mandate Creep»

  • Is management measuring and reporting on the right things?

    Lack of program logic model and associated measurement

    framework minimizes ability to objectively evaluate

    Recognize political realities