1 / 77

This project is co-financed by the European Union and the Republic of Turkey

Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey. This project is co-financed by the European Union and the Republic of Turkey.

rainer
Download Presentation

This project is co-financed by the European Union and the Republic of Turkey

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Bölgesel Rekabet Edebilirlik Operasyonel Programı’nın Uygulanması için Kurumsal Kapasitenin Oluşturulmasına Yönelik Teknik Yardım Technical Assistance on Institutional Building for the Implementation of RCOP in Turkey This project is co-financed by the European Union and the Republic of Turkey MONITORING & EVALUATIONfor RCOP Ankara, 12 – 16 December 2011 Monitoring & Evaluation, 12-16 Dec 2011

  2. This project is co-financed by the European Union and the Republic of Turkey SECTION I.MONITORING VSVALUATION • Clarification of definitions Monitoring & Evaluation, 12-16 Dec 2011

  3. Distinction between evaluation, monitoring and audit This project is co-financed by the European Union and the Republic of Turkey • Evaluation • Assessment of the efficiency, effectiveness, impact, relevance and sustainability of aid policies and actions • Monitoring • Ongoing analysis of programme/project progress towards achieving planned results with the purpose of improving management decision making • Audit • Assessment of • the legality and regularity of project expenditure and income i.e. compliance with laws and regulations and with applicable contractual rules and criteria; • whether project funds have been used efficiently and economically i.e. in accordance with sound financial management; • whether project funds have been used effectively i.e. for purposes intended. • Primarily a financial and financial management focus, with the focus of effectiveness being on project results. Monitoring & Evaluation, 12-16 Dec 2011

  4. Evaluation-monitoring-project management This project is co-financed by the European Union and the Republic of Turkey • Monitoring: to beaware of the present situation • Evaluation: to rationallyconnect the present situation with the desired future situation • Project management: operations and transactions (contracting, approval of reports, payments…) but activities leading to these rely on monitoring and to (interim) evaluation outcomes • Potential overlaps of monitoring and project management activities • Segregation of duties, clear definition of responsibilities • Cooperation Monitoring & Evaluation, 12-16 Dec 2011

  5. Distinction between and evaluation monitoring This project is co-financed by the European Union and the Republic of Turkey Monitoring • What?Monitoring is an integral part of a day-to-daymanagement. • How?Monitoring embodies the regular tracking of inputs, activities, outputs, reach, outcomes, and impacts of development activities at the project program, sector and national levels • Why?Monitoring provides information by which management can identify and solve implementation problems and assess progress towards project's objectives Evaluation • What?Evaluation is an assessment that refers to design, implementation and results of completed or on-going project / program / policy. • How?Evaluation should be systematic and objective. Key criteria to be used are: relevance, developmental efficiency, effectiveness, impact and sustainability. • Why?Evaluation should provide credible and useful information to enable the incorporation of lessons Monitoring & Evaluation, 12-16 Dec 2011

  6. Distinction between and evaluation monitoring This project is co-financed by the European Union and the Republic of Turkey Monitoring • focused on daily management issues • typical questions: “How many?” "When?” “How?” “For how much?” • assess whether activities are implemented effectively and efficiently Evaluation • addresses strategic questions: “So what?”(impact and sustainability) and “Why?” (relevancy) • analysisis deeper and seeks for actual cause-results relationships • perception of “big picture" Monitoring & Evaluation, 12-16 Dec 2011

  7. Distinction between and evaluation monitoring This project is co-financed by the European Union and the Republic of Turkey Monitoring • usually means a system • data collected and analyzed more or less frequently • according to a predefine timetable (Performance Measurement Plan) • regularity and continuity of data collection • methodology used to analyze it Evaluation • flexibility in specifying, which aspects of the program should be assessed, when and how. Monitoring & Evaluation, 12-16 Dec 2011

  8. Distinction between and evaluation monitoring This project is co-financed by the European Union and the Republic of Turkey • Monitoring • Part of modern project management • Generates useful information for project manager • where are bottlenecks? • how are we doing towards our objectives? • are expenses under control? • Utilityis the primary feature • Evaluation • Target groups:donors, planners, assistance recipients and wider public • Have we achieved our goals? • Are our results sustainable? • Have we learned anything for the future? • Focus on • transparency of evaluator’s approach • revealing cause-effect relationshipsbetween subsequent layers of analysis. Monitoring & Evaluation, 12-16 Dec 2011

  9. Distinction between and evaluation monitoring This project is co-financed by the European Union and the Republic of Turkey • Monitoring • Project management needs  rapid assessment methods • fast feedback • not very expensive • Evaluation • Scrupulous research methodologies • representative surveys • comprehensive quantitative analyzes Monitoring & Evaluation, 12-16 Dec 2011

  10. M&E – Comparative characteristics This project is co-financed by the European Union and the Republic of Turkey Monitoring & Evaluation, 12-16 Dec 2011

  11. M&E system diagramme This project is co-financed by the European Union and the Republic of Turkey Evaluation Monitoring Monitoring & Evaluation, 12-16 Dec 2011

  12. This project is co-financed by the European Union and the Republic of Turkey SECTION II.EVALUATION • EU Regulations on evaluation • Introductory guide on basic notions Monitoring & Evaluation, 12-16 Dec 2011

  13. COUNCIL REGULATION 1085/2006 establishing IPA This project is co-financed by the European Union and the Republic of Turkey Article 22 Evaluation • Commission shall regularly evaluate • the results and efficiency of policies and programmes • effectiveness of programming whether the objectives have been met • Objectives of Evaluation • to enable the COM to formulate recommendations • results shall feed back into programme design and resource allocation Monitoring & Evaluation, 12-16 Dec 2011

  14. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey • Title II Common rules for implementation/Chapter VEvaluation and monitoring • Article 57 Evaluation • Objectives: to improve the quality of • effectiveness and consistency of the assistance • strategy and implementation of the programmes. • Subject: Multi-annual indicative planning documents (ex ante evaluation) • Forms: • Strategic • Ex ante • Interim • Ex post Monitoring & Evaluation, 12-16 Dec 2011

  15. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey Article 57 Evaluation (cont’d) • Interim: • At least one during implementation • Specifically when the monitoring reveals significant departure from the goals initially set. • Ex post • the responsibility of the Commission • Shall include identifiable IPA component-specific results • The results of ex ante and interim evaluation shall be taken into account in the programming Monitoring & Evaluation, 12-16 Dec 2011

  16. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey CHAPTER III/Implementation/Section 1 Framework for implementation and principles Article 78 Implementation principles in the event of participation in Community programmes and agencies (as amended by COM REGULATION 80/2010) • All evaluations shall be carried out by the COM prior to the conferral of management powers on the beneficiary country, • After the conferral • BC is responsible for interim evaluation; • COM’s rights to perform any ad-hoc evaluations remains; • ex-ante and ex-post remain with COM • BC’s right to carry out ex-ante and ex-post as it deems necessary. Monitoring & Evaluation, 12-16 Dec 2011

  17. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey Title III Regional development and human resources development/Chapter IIIImplementation/Section 3 Evaluation and monitoring Article 166 Evaluation • ex ante • Beneficiary countries - under the responsibilityof the operating structure. • Foreach operational programme separately. • If dulyjustified and in agreement with the COM BC may carry out a single ex ante evaluationcovering more than one operational programme. • Shall aim to • optimise the allocation ofbudgetary resources under operational programmes • improveprogramming quality • identify and appraise • thedisparities, gaps and potential for development, • the goals to beachieved, • the results expected, • the quantified targets, • Thecoherenceof the strategy proposed • and the qualityof the procedures for implementation, monitoring, evaluation • Annexed to OP Monitoring & Evaluation, 12-16 Dec 2011

  18. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey Article 166 Evaluation (cont’d) • Interim • During the programming period • Beneficiary countries • Linked to the monitoring of OPs, in particular where • this reveals asignificant departure from the goals initially set or • proposals are made for the revision of OPs • Results shall be sent to thesectoral monitoring committee for the OP and to COM • Evaluations shall be carried out by experts or bodies,internal or external, functionally independent of the authorities • The results shall be published Monitoring & Evaluation, 12-16 Dec 2011

  19. Purpose and principles This project is co-financed by the European Union and the Republic of Turkey • Logical elements of purpose of evaluation: • Assessment; • Systematic and objective; • Ongoing or completed project, programme or policy; • Design, implementation and results; • Determine the relevance and fulfillment of objectives, developmental efficiency, effectiveness, impact and sustainability; • Provide information that is credible and useful; • Enabling the incorporation of lessons learned into the decision-making process of both recipients and donors; Monitoring & Evaluation, 12-16 Dec 2011

  20. Purpose and principles This project is co-financed by the European Union and the Republic of Turkey • Principles of the approach to evaluation: • Impartiality and independence of the evaluation process from the programming and implementation functions; • Credibility of the evaluation, through use of appropriately skilled and independent experts and the transparency of the evaluation process, including wide dissemination of results; • Participation of stakeholders in the evaluation process, to ensure different perspectives and views are taken into account; • Usefulness of the evaluation findings and recommendations, through timely presentation of relevant, clear and concise information to decision makers. Monitoring & Evaluation, 12-16 Dec 2011

  21. Evaluation types This project is co-financed by the European Union and the Republic of Turkey Ex ante Article 2 of the Financial Regulation • Use of budget appropriations • principles of sound financial management • quantified objectives must be identified • „mobilisation of Community resources must be preceded by an evaluation to ensure that the resultant benefits are in proportion to the resources applied.” Monitoring & Evaluation, 12-16 Dec 2011

  22. Evaluation types This project is co-financed by the European Union and the Republic of Turkey Ex ante • Implementation Rules for FR – ex ante evaluation shall identify: • need to be met in the short or long term; • objectives to be realised; • results expected and the indicators needed to measure them; • added value of Community involvement; • risks, including fraud; • lessons learned from similar experiences in the past; • volume of appropriations, human resources and administrative expenditures; • monitoring system to be set up. Monitoring & Evaluation, 12-16 Dec 2011

  23. Evaluation types This project is co-financed by the European Union and the Republic of Turkey Ex ante • Process that supports the preparation of interventions • - related to programming ~ quality assurance • Purpose: to gather and analyse information • define objectives, • ensure that these objectives can be met, • that the instruments used are cost-effective, • reliable later evaluation will be possible • Independent from planners Monitoring & Evaluation, 12-16 Dec 2011

  24. Evaluation types This project is co-financed by the European Union and the Republic of Turkey Interim • Assessment of the quality of programme implementation based on the first outputs and results • relevance of the adopted strategy; • newly occured factors having an impact on the implementation; • objectives have been defined accurately; • indicators are relevant; • so-far effectiveness and extrapolation; • management quality of the project implementation; • reliable collected data, including the monitoring system • Independent from implementors Monitoring & Evaluation, 12-16 Dec 2011

  25. Evaluation types This project is co-financed by the European Union and the Republic of Turkey • Ex post • After programme completion • Examining long-lasting effects • impact (more) visible • verification to what extent objectives have been achieved • Sustainability – estimated character • Anticipated and unexpected effects • Source of information for future programming excercises Monitoring & Evaluation, 12-16 Dec 2011

  26. Major tasks for an Evaluation Unit This project is co-financed by the European Union and the Republic of Turkey • Identifying the need for an evaluation and selecting the topics/themes to be evaluated; • Designing the evaluation, including preparing the Terms of Reference; • Drafting tender documents for the evaluation study and selecting the contractor according to the established rules; • Briefing the contractor and the parties involved, and supporting the evaluation mission; • Ensuring the production of a high quality evaluation report and of the dissemination of evaluation findings and recommendations. Monitoring & Evaluation, 12-16 Dec 2011

  27. Tools and key documents This project is co-financed by the European Union and the Republic of Turkey Primary tools available to support a project evaluation: • Terms of reference for the evaluation mission; • The project’s Logframe matrix - to help assess what has been achieved against plan; • Programme planning documents; • Monitoring reports (internal and external); • IPA Interim Evaluation Reports; • IPA Monitoring reports • Minutes of SMCs and other organisations involved in monitoring Monitoring & Evaluation, 12-16 Dec 2011

  28. Key documents produced This project is co-financed by the European Union and the Republic of Turkey • Terms of Reference for the evaluation mission, • Final Evaluation Mission Report • should mirror the structure of the main evaluation criteria • taking into account • the nature of the project, • the stage at which the evaluation is carried out, • the users for whom the report is prepared. Monitoring & Evaluation, 12-16 Dec 2011

  29. Evaluation Criteria This project is co-financed by the European Union and the Republic of Turkey Relevance • @ significance; pertinence • To what extent do we do the right things ? Does it make sense? • ~ to the identified problems or real needs to be addressed • The appropriateness of project objectives to • problems that it was supposed to address • physical and policy environment within which it operated • The extent to which the objectives of a programme/project are consistent with beneficiaries’ needs, country needs, global priorities. Monitoring & Evaluation, 12-16 Dec 2011

  30. Evaluation Criteria This project is co-financed by the European Union and the Republic of Turkey • Relevance (cont’d) • Relates primarily to programme design • extent to which correctly address the identified problems or real needs. (Needs to be kept under review throughout the life of the project in case changes occur) • logic and completeness of the project planning process • internal logic and coherence of the project design • at two points in time: when the project was designed, and at the time of the evaluation. • Focus • identification of real problems or needs • correct beneficiaries Monitoring & Evaluation, 12-16 Dec 2011

  31. Evaluation Criteria This project is co-financed by the European Union and the Republic of Turkey • Efficiency • How well the various activities transformed the available resources into the intended results (outputs) • quantity, quality and timeliness. • value-for-money: similar at lower cost in the same time • Focus: the quality of day-to-day management, i.e: • budget • personnel, information, property, etc; • risk and change • relations/co-ordination with actors • time (deadlines) Monitoring & Evaluation, 12-16 Dec 2011

  32. Evaluation Criteria This project is co-financed by the European Union and the Republic of Turkey • Effectiveness • Contribution made by results to achievement of the Project Purpose • Benefits accruing to target groups • Did we achieve our objectives? To what extent did our outputs produce the desired outcomes? • By how far the intended beneficiaries really benefited from the products or services it made available? • Focus: • whether the planned benefits have been delivered and received; • appropriateness of the indicators of benefit used; • how unplanned results may have affected the benefits received; Monitoring & Evaluation, 12-16 Dec 2011

  33. Evaluation Criteria This project is co-financed by the European Union and the Republic of Turkey Impact • Effect of the project on its wider environment, and its contribution to the wider policy or sector objectives (as summarised in the project’s Overall Objective). • Outcome: relationship between the project’s purpose and overall objectives • Benefits received by the target beneficiaries had a wider overall effect on larger group • Focus: • to what extent the planned overall objectives have been achieved, • how far that was directly due to the project; • unplanned impacts’ effects on overall impact; • Ex: higher standard of living, increased food security, democratic rule of law Monitoring & Evaluation, 12-16 Dec 2011

  34. Evaluation Criteria This project is co-financed by the European Union and the Republic of Turkey Sustainability • Continuation of benefits, effects generated by a programme/project after its termination. • Likelihood of benefits produced to continue to flow after external funding has ended, • Focus: • Ownership • Policy support and the responsibility of the beneficiary institutions • Institutional capacity, commitment • Economic and financial factors, socio-cultural aspects, gender equality • Technology, environmental aspects • Financial sustainability • Ex: a micro-credit scheme that is generating enough money for the scheme to operate, cover risks and develop its staff Monitoring & Evaluation, 12-16 Dec 2011

  35. A Quick reflection This project is co-financed by the European Union and the Republic of Turkey • Appropriateness is not about money. It is about what was achieved against what was needed. • Effectiveness is not about money. It is about how much it was achieved. • Efficiency is about how much was achieved and how much money was spent. • Sustainability is not about money. It is about the embedment of what was achieved in the given context (natural, societal, economic…) Monitoring & Evaluation, 12-16 Dec 2011

  36. Decision options This project is co-financed by the European Union and the Republic of Turkey • Look out: • one may be very efficient but not effective! • or very effective but not relevant! • to do things right is fine, but to do the right things is for sure far more important! Monitoring & Evaluation, 12-16 Dec 2011

  37. Decision options This project is co-financed by the European Union and the Republic of Turkey • Continue project implementation as planned • Re-orient/restructure the project • To stop the project (mid-term evaluation); • Modify the design of future projects or programmes in light of lessons learned (ex-post evaluation) • Modify policies, co-operation strategies, and subsequent programming or identification exercises (sector, thematic or cross-sector evaluations) Monitoring & Evaluation, 12-16 Dec 2011

  38. Performance Ratings This project is co-financed by the European Union and the Republic of Turkey • Performance rating in two steps: • Numerical rating on each criteria for each component of the sector from –2 to +2: • Conversion of numerical overall rating into a qualitative rating. • Qualitative ratings • « Highly satisfactory » The programmes reviewed are expected to achieve or exceed all the objectives set during their lifetime; most numerical ratings in the range of 1 to +2 • « Satisfactory » The programmes reviewed are expected to largely achieve the objectives; most numerical ratings in the range of 0 to 1 • « Unsatisfactory » The programmes reviewed are not expected to achieve most of the objectives; most numerical ratings in the range of –1 to 0 • « Highly unsatisfactory » The programmes reviewed are not expected to achieve any of the objectives; most numerical ratings in the range of -1 to –2 Monitoring & Evaluation, 12-16 Dec 2011

  39. Quality Assessment of IE Reports This project is co-financed by the European Union and the Republic of Turkey • Criteria – Rate – Remarks • A. General: Does the report design appropriately fit the evaluation? • B. Sound sectoral overview: to what extent are the sector composition and priorities appropriately described? • C1. Sound analysis: to what extent are the facts and data adequately analysed? • C2. Sound analysis: to what extent have the indicators of achievement been adequately considered and have they been used properly where was possible? • D. Robust Findings in the implementation evaluation: do the Conclusions follow logically from, and are they justified by, the data described in the Sectoral Overview? Monitoring & Evaluation, 12-16 Dec 2011

  40. Quality Assessment of IE Reports This project is co-financed by the European Union and the Republic of Turkey • E. Impartial conclusions: does the report provide value judgements based upon the five evaluation criteria of relevance, efficiency, effectiveness, sustainability and impact? • F. Useful recommendations: to what extent do the Recommendations follow logically from the Conclusions? Are they operational? Do they clearly address the monitoring sector and are they targeted to the different stakeholders? • G. The executive summary: to what extent is the executive summary a synthesis and does it meet the requirements set out in the template guidelines? • H. Annexes: to what extent do the Annexes support the analysis in the main text? • I. Overall style, structure and text design: within the template’s framework, to what extent is the text easily readable and accessible to the various categories of readers so that the main messages are easily detectable? Monitoring & Evaluation, 12-16 Dec 2011

  41. Quality Assessment of IE Reports This project is co-financed by the European Union and the Republic of Turkey Taking into account the contextual constraints on the evaluation, the overall quality rating of the report is considered to be: Monitoring & Evaluation, 12-16 Dec 2011

  42. Why Results Are Ignored? This project is co-financed by the European Union and the Republic of Turkey • Implications may not be presented in a way that nonresearchers can understand. • Results sometimes contradict deeply held beliefs. • Vested interest in a program. Monitoring & Evaluation, 12-16 Dec 2011

  43. This project is co-financed by the European Union and the Republic of Turkey SECTION III.MONITORING • EU Regulations on monitoring • Monitoring system and its evaluation Monitoring & Evaluation, 12-16 Dec 2011

  44. MONITORING This project is co-financed by the European Union and the Republic of Turkey • @ follow up , controlling • A continuing observation • uses systematic collection of relevant and selected data • to provide management and the main stakeholders of a programme/project with indications • of the extent of progress and achievement of objectives • of process and impact. Monitoring & Evaluation, 12-16 Dec 2011

  45. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey • Title II Common rules for implementation/Chapter VEvaluation and monitoring • Article 58 Monitoring in the case of decentralised management (IPA Monitoring Committee) • BC in agreement with NIPAC and COM • to ensure coherence and coordination in the implementation of the IPA components • meeting the objectives set out in the financing agreements and MIIFP • based on the elements given by the SMCs Monitoring & Evaluation, 12-16 Dec 2011

  46. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey IPA Monitoring Committee – proposals • to COM, NIPAC and NAO for: • any actions to ensure the coherence and • crosscomponent corrective measures needed to ensure the achievement • to SMCs for: • decisions on any corrective measures to ensure the achievements of programme Monitoring & Evaluation, 12-16 Dec 2011

  47. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey IPA Monitoring Committee – organisation • Internal rules of procedure • in compliance with a monitoring committee mandate established by the COM • within the institutional, legal and financial framework of the BC • Members: Commission, NIPAC, NAO, OS, strategic coordinator • COM and NIPAC co-chair • Meetings at least once a year • Intermediate meetings on a thematic basis. Monitoring & Evaluation, 12-16 Dec 2011

  48. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey Article 59 Sectoral monitoring committees in the case of decentralised management • Assist IPA Monitoring Committee • Attached to programmes or components. • May include representatives of civil society, • Objective: • effectiveness and quality of the implementation • in accordance with the related sectoral and/or financing agreements. • Proposals • to the COM and NIPAC, cc. NAO • for decisions on any corrective measures to ensure the achievements of programme objectives Monitoring & Evaluation, 12-16 Dec 2011

  49. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey Sectoral monitoring committee - reports • to IPA monitoring committee on • progress • by priority axis/ measures or operations; • results achieved, financial implementation indicators • any aspects of the functioning of the management and control systems raised by the audit authority, NAO or the competent accrediting officer Monitoring & Evaluation, 12-16 Dec 2011

  50. COM REGULATION 718/2007implementing 1085/2006 This project is co-financed by the European Union and the Republic of Turkey Article 61 Annual and final reports on implementation • Sectoral annual report and sectoral final report by OS • Sectoral annual report: financial year • Sectoral final report: whole period of implementation • SMC examination  to be sent to NIPAC, NAO and COM Monitoring & Evaluation, 12-16 Dec 2011

More Related