Assessment of the Evaluation Culture

1. Assessment of the Evaluation Culture Approach

2. Introduction • PM&E project - Scope of the work • Evaluation Culture Assessment • Why do we need it? • Agreement on the methodology

3. Topics of Discussion • Evaluation - background • What was already achieved in Romania • Further steps taken under the present TA contract • Discussion and agreement on the methodology to be employed in assessing the evaluation culture in Romania

4. Evaluation - background • Evaluation Advisory Group (EAG) • “birth date” - 25 July 2002 • composition • mandate • first meeting - 27 March 2003

5. Evaluation - background • Objectives of EAG • To promote the development of medium-term National Strategies for building local monitoring and evaluation capacities (which would also cover the national public funds) • To support candidate countries (via exchange of experience) in their preparation of draft short-term Action Plans for development of local monitoring and evaluation capacities within the framework of decentralisation of monitoring and evaluation. • To prepare a Guide to Good Practices in Monitoring and Evaluation Capacity Building, including national strategies, monitoring and evaluation models, capacity building strategies and means to boost monitoring and evaluation capacity.

6. Evaluation - background • Content of the GUIDE • Monitoring and evaluation models in Member States and the Candidate Countries - Key output: Establishing a typology of good practices of national strategies for monitoring and evaluation. • National Strategies - Key output: Establishing a typology of good practices of national strategies for monitoring and evaluation. • Local capacity building - Key output: A survey of good practices in evaluation capacity building.

7. Evaluation - background • EAG-WG1 would research and report on National Strategies and Monitoring and Evaluation Models in order • to develop and implement an approach which identifies current good evaluation practice in Candidate Countries and Member States • to provide guidance on the implementation of good practice to assist Candidate Countries individually and collectively • to recommend contributions for the Guide • EAG-WG2 would research and report on the development of a Capacity Building Strategy in order • to develop and implement an approach to identify and record good practice in evaluation capacity building strategy in Candidate Countries and Member States • to provide guidance on the implementation of good practice to assist Candidate Countries individually and collectively • to make recommendations to the EAG for consolidation within a handbook of good practice.

8. Evaluation - background • Main conclusions regarding the situation in candidate Countries - WG2-July&Nov 2003 - based on a questionnaire completed and submitted by each country • The evaluation function was likely to be a small, under-resourced unit; • The main identifiable impact reported has been from the Phare Programme evaluation; • Outside Commission requirements, evaluations are rare; • Evaluations do not yet make a significant impact on accountability; • Institutionalising factors (for example, regulation) are poorly developed; • In the short term, evaluation activity will remain centralised; • Candidate Countries have anticipated the need for training; • Programme/ project monitoring is focused on the Commission’s Phare IE process.

9. Evaluation - backgroundMilestones • Representatives of several Candidate Countries were able to make presentations of their National Strategies in November 2003 • The Guide to good practices of evaluation capacity building was issued in draft by EMS on 15 January 2004 • Two issues remain for most Candidate Countries: the political will and national evaluation capacity, and the two are interrelated.

10. What was already achieved in Romania • ACTION PLAN FOR THE TAKE OVER OF INTERIM EVALUATION FUNCTION FOR PHARE PROGRAMME- First Pillar of Romanian National Evaluation System – drafted and presented in February 2004 • On - going developments

11. Further steps taken under the present TA contract • ECU - institutional building/strengthening • Awareness rising on evaluation • Assessment of the legal framework related to evaluation • Assessment of the evaluation culture • Elaboration of the National Strategy for Evaluation • Capacity building on evaluation supply side • preparation of further interventions

12. Evaluation Culture • “The evaluation process is now seen as a major contributor to state accountability and good governance, but its institutionalisation has been slow, with a few exceptions. One of the obstacles to this is a poorly developed evaluation culture: a practice of asking questions, demanding answers and expecting change.”

13. Evaluation Culture The nine empirical criteria which must be satisfied before an evaluation culture can be said to be fully established: • I Evaluation takes place in many policy domains • II Supply of domestic evaluators in different disciplines • III National discourse concerning evaluation • IV Professional organisations • V Degree of institutionalisation – Government • VI Degree of institutionalisation – Parliament • VII Pluralism of institutions or evaluators performing evaluations within each policy domain • VIII Evaluation within the Supreme Audit Institution • IX Proportion of outcome evaluations in relation to output and process evaluations

14. Evaluation Culture For Candidate Countries, three further criteria could be added to those identified above: • X Monitoring capacity • XI Evaluation Diversity: Strategy, Policy, Programme, Projects • XII Information flow within government relating to evaluation These are medium term goals against which Candidate Countries interested in developing an evaluation culture should attempt to benchmark themselves.

15. Drafting of NES Strategy document • Completion of Evaluation Culture Assessment – May 19 • First draft of NES – May 22-June 30 • Presentation of first draft to EWG – July 25 • Presentation of final draft to EWG – August 22. • Launch of NES – Evaluation Conference to take place in September. • Regional road-trip – 8 NES-promotional events to take place in the regions, September-October.