Guidelines on the Mid-term Evaluation

1. Guidelines on the Mid-term Evaluation John Grieve & Hannes Wimmer Evaluation Helpdesk Evaluation Expert Committee Meeting

2. Content • Overview of the Mid-Term Evaluation (MTE) • Main steps of the MTE • Tools to support the MTE Evaluation Expert Committee Meeting

3. Overview: legal provisions • Article 86 of Council Regulation 1698/2005 specifies the management and the functions relating to the MTE as follows: • “In 2010, ongoing evaluation shall take the form of a separate mid-term evaluation report. • That mid-term evaluation shall propose measures to improve the quality of programmes and their implementation. • A summary of the mid-term evaluation reports shall be undertaken on the initiative of the Commission.” drafting of Terms of Reference? selection of evaluators? organisation of tender procedures? use of evaluation methodologies? Evaluation Expert Committee Meeting

4. Purpose of the guidelines • support Member States and the relevant national or regional authorities to frame the work of their independent evaluators • providing further guidance on key aspects of the MTE Evaluation Expert Committee Meeting

5. Overview: main characteristics (1/2) • is a specific part of ongoing evaluation activities (=combination of ongoing + MTE-specific activities) • has a critical role as formal mid-point review to inform on adjustments of the current programmes  importance of results and impact level! Evaluation Expert Committee Meeting

6. Overview: main characteristics (2/2) • draws on ongoing evaluation activities • intervention logic and target levels from ex-ante • data collection and monitoring • thematic studies, Community guidance documents, Synthesis ex-ante, previous programmes • contributes to the ex-post  The MTE is embedded into the continuous process of ongoing evaluation, which provides the solid ground for it! Evaluation Expert Committee Meeting

7. Challenges of the MTE • RDPs at an early stage in implementation • availability of data • impacts emerge late • timing and organisation The MTE should not be reduced to issues concerning data-availability: The QUESTIONS that MAs would like answered are crucial! Evaluation Expert Committee Meeting

8. Main steps of the MTE PREPARATION • review of evaluation questions & related indicators • information gathering • setting up of the steering group  • establishment of the evaluation mandate  • establishment of the ToR • ... recommended step! IMPLEMENTATION • independent and interactive MTE process • managementandfeedback procedures, MA and evaluator • assessment of quality of the MTE report DISSEMINATION • dissemination and use of results • - follow-up of MTE Evaluation Expert Committee Meeting

9. STEP 1: Review of evaluation questions and indicators • CMEF: • For setting-up the evaluation system, it is inevitable for the managing authority to review the common and programme-specific evaluation questions and the related indicators in order to assess what needs to be done in terms of information gathering and analysis in order to answer these questions in a meaningful and appropriate manner. […] Guidance note B, chapter 5.1.3, p.9 Evaluation Expert Committee Meeting

10. STEP 1: Review of evaluation questions and indicators • Key questions: • Do the CEQ and indicators capture the full range of results and impacts or is there a need for further fine-tuning? • Are they sufficiently consistent with the CMEF to allow a European synthesis? • Are the programme specific evaluation question and indicators well defined to address the RDP’s specificities? Use the programme’s ex-ante evaluation findings regarding the intervention logic and measures. Justify any variations from the CMEF. Define further additional questions or indicators where you identify gaps! Evaluation Expert Committee Meeting

11. STEP 2: Information gathering • CMEF • MA ensures that evaluators have sufficient data on • general trends • outputs • results • in order to enable evaluators to assess the impacts Evaluation Expert Committee Meeting

12. STEP 2: Information gathering Use evaluation questions and indicators as checklist! • Key questions: • - Does the MA provide evaluators with sufficient data on general trends, outputs, results to enable them to assess the impacts? • Has it been clarified what data is collected by the MA and what by evaluators? • Have the additional data-needs in light of the programmes’ specificities been considered? • Is all the relevant information from ongoing evaluation activities readily available? Clarify where data is recorded and the accessibility/usability of information. . Ensure support for evaluators who undertake interviews or surveys! Consider date of availability of studies, data from third parties etc. Evaluation Expert Committee Meeting

13. (STEP 3): Setting up of steering group  • CMEF: • In order to ensure a high quality of the evaluation, a regular consultation of stakeholders should be ensured. • The setting up of a steering group which accompanies the evaluation process and which involves representatives of different departments is advisable. • The steering group should contribute to the preparation of the terms of reference.  evaluation mandate! • The members of the steering group can provide access to additional information; they shall support and monitor the work of the evaluator. Guidance note B,p. 9 Evaluation Expert Committee Meeting

14. (STEP 3): Setting up of steering group  Adds value to evaluation findings and their follow-up • Key questions: • Has a steering group been set-up for the purpose of the MTE? • What is the composition of the steering group? • Does the composition of the SG (i.e. the field of expertise of the members) properly reflect the focus of the programme ? • Have the roles and activities of the SG been clarified? MA, paying agency, measure responsibles, evaluation unit, sub-regional authorities, LAGs…(N.B. no final beneficiaries) Ensure that technical expertise reflects content of RDP ! Evaluation Expert Committee Meeting

15. (STEP 4): Establishment of the evaluation mandate  • CMEF • […] In order to allow a target-oriented preparation of the terms of reference in the steering group, it is advisable to establish an evaluation mandate, giving an overview on context, scope, timing, and objectives of the evaluation in question. Guidance note B, p.9 Evaluation Expert Committee Meeting

16. (STEP 4): Establishment of the evaluation mandate  Use evaluation mandate as guide for the preparation of ToR and the work of the SG. • Key questions? • Has the MA or SG specified the context, scope, timing, and objectives of the MTE in the form of an “evaluation mandate”? • Who has been involved in the preparation of the evaluation mandate? • Has the evaluation mandate been acknowledged by all involved bodies? (MA, SG, evaluators etc.) MA or evaluation unit should initiate preparation of mandate. Involve the SG in order to ensure consistency. Evaluation Expert Committee Meeting

17. STEP 5: Establishment of the ToR • CMEF • The terms of reference serve the purpose of establishing the framework for evaluation activities during the different phases of ongoing evaluation (ex-ante evaluation, mid-term evaluation, ex-post evaluation). • In addition, they form the basis for calls for tenders or the establishment of service requests within framework contracts. Guidance note B, chapter 5.1.2., p.9 Evaluation Expert Committee Meeting

18. STEP 5: Establishment of the ToR Ensure that previous steps are undertaken (review of questions, information gathering etc.) • Key questions: • Has a proper framework for the MTE (incl. as part of the ongoing evaluation activities) been established? • Have results from ongoing evaluation activities been properly fed into the ToR? • Has an appropriate timetable been defined (incl. deadlines, meetings etc.)? • Do the ToR properly reflect the evaluation requirements? Ensure that findings from ongoing evaluation fed into ToR. Cross check with outline-structure . Evaluation Expert Committee Meeting

19. STEP 5: Establishment of the ToR framework, legal requirements, purpose • Indicative OUTLINE for technical specifications for the Terms of Reference: • (a) Context of the MTE • (b) Scope of the MTE • (c) Objectives of the evaluation • (d) Evaluation questions • (e) Tasks to be performed • (f) Timing and content of deliverables • (g) Organisation of work • (h) Sources and documentation description of measures, geographical area, period, specific focus common + programme specific objectives structuring, observing, analysing, judging budget, responsibilities, interaction with SG Evaluation Expert Committee Meeting

20. STEP 5: Further considerations Ensure that ToR reflect scale and scope of the programme and the required evaluation activities. • Key questions: • Are ongoing evaluation activities (and MTE) adequately resourced to be effective? • Is the concept of the independence of evaluators respected? • Is a regular consultation with stakeholders ensured, are the interactions clarified? • Does the timing for launching/closing the procedures allow time for submitting the final report to the EC within the deadlines? Bodies without direct involvement in the implementation, management and financing of the programmes. Ensure sufficient time for preparation, primary research, interaction with the steering group…. Evaluation Expert Committee Meeting

21. IMPLEMENTING the MTE Provide for an effective inception process with an accurate project management plan. • Key questions: • What does an independent but interactive evaluation process look like? • Is an ongoing dialogue between evaluators, steering group and MA ensured? • How can the quality of the final MTE Report be guaranteed? • Can the final MTE-Report be submitted in time to the MC and the EC? Do not leave the evaluator alone! Provide for iterative and regular exchanges with the steering group. Anticipate a quality assessment of the final MTE-report by the steering group. Evaluation Expert Committee Meeting

22. DISSEMINATING the results • Key questions: • How can an effective dissemination process be planned? • What are appropriate means for dissemination? Define from the outset the purpose, target audiences, dissemination tools, timing and resources. Targeted presentations, conference contrib., publications, case studies, extracts, summaries, thematic papers, newsletters, websites, discussion forums. Evaluation Expert Committee Meeting

23. … 2009 2010 2011 … PREPARATION IMPLEMENTATION DISSEMINATION Review Common & Specific Evaluation Questions Independent and Interactive Evaluation Process:structuring, observing analysing, judging Disseminationof Evaluation Results through various means (presentations, contributions to publications, articles, websites..)to different Target Groups. Data Collection and Information Gathering Establishing the steering group Ongoing Dialogue between Evaluator, SG, MA Evaluation Mandate Quality Assessment of Final Report by SG Drafting the Terms of Reference Submission Final MTE- Report to MC & EC Timeline Evaluation Expert Committee Meeting

24. Helpdesk-Tools for the MTE • Survey on preparation of MTE-results • Guidelines on preparation of the MTE • Collection of FAQ on Mid-Term-Evaluation • Good Practices on MTE-preparation process etc. • PPT on specific issues (e.g. ongoing evaluation, MTE-preparation etc.) Evaluation Expert Committee Meeting

25. Thank you for your attention! Evaluation Expert Committee Meeting