Evaluation Presentation to the Brazilian Conference November 23, 2004

1. Evaluation Presentation to the Brazilian ConferenceNovember 23, 2004

2. Overview • Agriculture in Canada • Department of Agriculture and Agri-Food • Evaluation Mandate and Profile • Evaluation Planning • Community Pastures Program • Program Overview • Evaluation Framework • Evaluation Findings • Lessons Learned • Questions

3. Agriculture in Canada

4. Agriculture in Canada • Accounts for some $112 billion in annual retail and food service sales • Is $81 billion of total economy • Represents about 8% of the country's total Gross Domestic Product • Is world's third-largest agri-food exporter behind the United States and the European Union • Contributes some $7 billion to Canada's annual trade balance, represents about 10 % of the total Canadian trade surplus • Provides 1 of every 8 jobs

5. Department of Agriculture and Agri-Food

6. Agriculture and Agri-Food 2002-2003 Expenditures

7. Agriculture and Agri-Food Portfolio Organizational Chart

8. Department located across Country

9. Evaluation Mandate and Profile

10. Audit and Evaluation Mandate Provide the Departmental Audit and Evaluation Committee with independent assurance, information and advice on the effectiveness, efficiency and economy of programs, policies and operations

11. Audit and EvaluationOrganization Frank Brunetta Director General Audit and Evaluation Brigitte Marois Director Financial Audit Richard Hill Director Planning and Coordination Rosemary Stephenson Director Audit Eric Seraphim A/Director Evaluation Grants and Contributions Revenue Expenditures Special Investigations Central Agency Coordination Advice and Assessements Follow-up Planning and Development Information Management Technology Special Investigations Evaluations

12. Audit and Evaluation Profile Mission Excellence Through Assurance and Information Key Result Commitments Provide DAET: Assuranceon the soundness of risk management strategies and practices, the design and operation of management control frameworks, systems and practices, and information used for decision making and reporting; Informationon the performance of departmental policies, programs and initiatives which is timely, strategically focussed, objective and evidence-based. • Our Services • Our key Result Commitments are delivered through a comprehensive range of services: • planning and conduct of independent audits, evaluations and other reviews approved by the DAET and/or Treasury Board • follow-up on management action plans developed in response to audit and evaluation recommendations; • advice and consultation on various departmental priority areas; • advice on the design of manager-led reviews; • management consulting and problem solving support; • coordination of Central Agencies activities within the department; and • special examinations as directed by the Deputy Minister (DM) Accountability While the risk-based planning and reporting of our audits, evaluations and assessments is approved by DAET, we are accountable to the DM for managing our function in accordance with the TB policies. Collaboration for Excellence We view and treat our work as an integral part of the departmental management process. Our approach is therefore grounded in a collaborative service-oriented philosophy. Departmental managers can help us help them succeed by supplying appropriate, accurate and timely information and ensuring we understand their needs and expectations.

13. Audit and Evaluation Profile (cont.) • Our Service Standards • Our work is conducted in a planned, systematic manner while adhering to established standards. • We measure our service performance on four factors: Quality, Timeliness, Value and Professionalism. • We ensure that staff or contract personnel undertaking reviews are trained and conversant with the standards and codes of ethics appropriate for the work being done and that our work is conducted in compliance with TB as well as other applicable industry standards: • The Institute of Internal Auditors (IIA) • Canadian Evaluation Society (CES) • Information Systems Audit and Control Association (ISACA) Client Survey Language Services are provided in both official languages. Our Costs In order to ensure we are able to deliver our mandate, our budget is focussed on projects approved by DAET. Requests not approved by DAET are performed if time permits and on a cost recovery basis. Our Service Guarantee We guarantee our work by paying or reimbursing for the hiring of a consulting firm to complete work which does not meet the criteria outlined in the approved Terms of Reference. Accessibility of our Reports Once completed and approved, reports, management responses and/or action plans are available on our Web site http://www.agr.gc.ca/review/rb-ep_e.php?page=list98and accessible by the public in accordance with the Government Audit and Evaluation Policies and the Access to Information and Privacy Acts. Questions or Comments Our primary point of contact is Frank Brunetta, Director General, Audit and Evaluation Team at (613) 759-6471, by facsimile at (613) 759-6499, or by E-Mail: brunettaf.agr.gc.ca

14. Audit and Evaluation Budget 2003-2004 Total Budget: $4,700,000 Full-time Equivalents: 27 $210K $125K $366K $402K $139K $1.6M $990K $162K Note: Indirect Costs includes items such as salaries associated with management and administration of functions, equipment, materials, and supplies.

15. Preliminary Project Identification Assess Projects Against Risk Factors Consult Senior Management, BoDs, Legal and the OAG (validate issues, priorities and projects) Prepare Recommended Audit & Evaluation Plan Resource Allocation (costing, HR capacity, scheduling) DAET Table Plan for Approval – April 14 The Risk-Based Strategically Focused Planning Process Environmental Scan Government Priorities Central Agency Priorities AAFC Strategic Outcomes Corporate Risks • Materiality • Federal/AAFC priorities • Sensitivity • Complexity • Management concern • Recent change • Results of prior work • Time since last audit/evaluation • Significance to strategic objectives • Interdependencies Risk Assessment DM Priorities

16. Evaluation Planning

17. Evaluation Framework Achievement of Results (including secondary impacts) The degree to which results have been achieved as well as any secondary effects observed (intended or unintended) that fall out of the iniatives. Relevance, Appropriateness and Acceptance Whether there is a continuing need, as well as the extent it meets the needs of all its stakeholders/partners. whether or not the organizational design (delivery mechanisms), terms and conditions and resources are sufficient to meet the objectives and its intended targets. Costs/Benefits The comparison of costs (direct and indirect) to the benefits (tangible and intangible) to illustrate the relationships among costs, inputs and outputs and outcomes (benefits). Management Effectiveness Insuring management commitment to manage, plan, organize, deliver, monitor, make adjustments, communicate and report on performance is clearly defined, understood and consistently implemented through appropriate policies, processes, practices and systems * adapted from the Canadian Comprehensive Audit Foundation (CCAF) Governance Checklist

18. Associated Framework Questions Achievement of Results over a period of time (including secondary impacts) Q: In what manner and to what extent were appropriate results achieved due to the program? Q: What client benefits and broader outcomes, both intended and unintended, resulted from carrying out the program? Q: In what manner and to what extent does the program complement, duplicate, overlap or work at cross purposes with other programs or organizations within this sector? Relevance, Appropriateness and Acceptance Q: To what extent are the objectives and mandate of the program still relevant to the future direction and needs of the sector and consistent with government priorities? Q: Are activities and the operational work plan consistent with the program mandate and plausibly linked to the stated results and other intended results?

19. Associated Framework Questions Relevance, Appropriateness and Acceptance (cont.) Q: Are there more cost-effective ways of delivering the existing program to achieve the stated objectives? What is the usefulness of the current delivery mechanism (are there any alternatives)? Q: Are clients and other stakeholders satisfied with the program’s activities? Are there formal and informal mechanisms to provide feedback? Costs/Benefits Q: What are the costs associated with the program and its various project components, e.g. key activities and their results (outputs, direct impacts)? Q: What are the benefits to society, the province, the sector, participants and governments? Q: Are the costs in line with the benefits? How do they compare to the expected vs actual results? Q: Are funds being distributed at the lowest possible cost?

20. Program Evaluation Methodology Typical approaches include: • Review of monitoring data from logic model or performance framework • Examination of reports and documentation • Statistical analysis • Benchmark analysis • Economic impact (quantitative) • Interviewing stakeholders • Etc.

21. Program Evaluation Methodology Minimize data bias by ensuring that for each evaluation issue, there are several independent lines of evidence related to that issue

22. Community Pastures Program

23. Community Pastures Program Overview • Land and homestead granting responsibility transferred from federal to provincial in 1930 • Significant amount of fragile land was homesteaded and cropped • Prolonged drought of the 1930s resulted in serious soil erosion problems • Resulted in passage of the Prairie Farm Rehabilitation Act in 1935 • Act: “ ...secure the rehabilitation of the drought and soil drifting areas...and to develop and promote within those areas, systems of farm practice, tree culture, water supply, land utilization and land settlement that will afford greater economic security...through the development, construction, promotion, operation and maintenance of any project or scheme...or agreements with any province, municipality or person...”.

24. Community Pastures Program Overview • Program established in cooperation with Saskatchewan and Manitoba. Alberta developed own program • Program governed by a series of Federal-Provincial agreements with Saskatchewan (1938, 1949, 1964, 1966, 1978) and Manitoba (1939, 1964, 1978) • Agreements establish legal authority and define terms and conditions: • Provinces acquire and place lands under federal control • Canada responsible for the development, operation and maintenance • Land base involves 2.3 million acres of erosion-prone marginal crown land

25. Original Program Objectives • “Land Utilization Program” established in 1937 • Permanent withdrawal of submarginal prairie land from cultivation • Development of such areas for grazing purposes; and • Resettlement on suitable farm land of farmers removed from submarginal areas

26. Current Program Objectives • “Community Pastures Program” redefined in 1978 • To make possible the removal of lands from unsuitable or unacceptable land use and to facilitate improved land use through their rehabilitation, conservation and management; and • To utilize the resource primarily for the summer grazing of cattle while assisting in stabilizing small farms and providing breeding bulls to encourage high quality, long term cattle production

27. Community Pastures Evaluation Framework Sample of Existing Matrix Review Framework

28. Evaluation Findings • 85 pastures, 24 in Manitoba; 60 in Saskatchewan; and 1 in Alberta • 80% of pastures surveyed (during1982-98) were in good to excellent condition (target of “at least 75%”) • Another approximately 3.2 million (ha) class 4,5 & 6 currently provincial jurisdiction and in production • Grazing service to approximately 225,000 head belonging to 3,500 patrons • 225,000 head represents 6 - 7% of cow-calf sector in Saskatchewan and Manitoba • 99-00 approximately $15 million direct costs to AAFC (185 FTEs) • Of $15M, $12 million cost-recovered (net cost of $3 million)

29. Direct and Indirect Benefits/Findings

30. Observations Stated Program Objective #1: To make possible the removal of lands and to facilitate improved land use • No significant increase of land base since the 1970's • Additional 3.2 million (ha) of class 4,5,6 marginal land under provincial jurisdiction and in crop production Stated Program Objective #2: To utilize the resource primarily for the summer grazing • 3,500 patrons utilize grazing service • Not targeted ? • Impact to income ? • 6-7% of cow calf sector in Saskatchewan & Manitoba ? • Patron turnover ratio ?

31. Observations World class expertise • Impressive and critical body of knowledge and expertise in land management, rehabilitation and sustainable practices CPP is achieving conservation & maintenance goals on existing pastures Program land base has not grown since 1970's • Despite the emphasis on the removal of land from unsuitable land use in this objective, there has been no significant removal of land from unsuitable use or any significant increase in the land base managed by CPP since the mid 1970's Role of Department and federal government • The primary focus of the program is pasture and grazing management, there is a need for a policy debate to determine if the Department should be directly involved in the day-to-day operations of community pastures Insufficient program information • There is insufficient performance information to accurately assess farm income and patron net worth impacts

32. Recommendations • A policy debate be initiated to determine the appropriateness of the Department managing pastures and grazing services to achieve environmental and farm income goals or whether more effective means are available • Explore ways to further apply CPP knowledge and expertise to the 3 mil ha of marginal land which continues to be cropped, as well as on the 10 mil ha of rangeland in less than good condition • Develop performance indicators to measure progress toward converting marginal land to improved land use, as well as to provide assurance that progress is being made toward farm income goals

33. Lessons Learned from Community Pastures Evaluation Focus on original mandate • What was program established to accomplish? • What were the intended results? • Are they still relevant? Look for «Program Mandate Creep» • Is management measuring and reporting on the right things? Lack of program logic model and associated measurement framework minimizes ability to objectively evaluate Recognize political realities

34. Questions?