Working Together on Impact Evaluation

1. Working Together on Impact Evaluation NONIE Subgroup 5 DFID, FAO, UNDP,WHO The Hague, 25 May 2007

2. The remit of Sub-group 5 An ultimate purpose of NONIE is to ensure a more substantial program of �impact evaluations� is carried out in the future, preferably in a coordinated or joint manner. This subgroup will make proposals for both the means by which this program should be undertaken and the content of the program (or at least means for identifying content).

3. Group plan of work � so far Take stock (again!): list of evaluation studies for four agencies Identify elements for joint working Propose scenarios

4. Group plan of work � next steps Build upon work of other NONIE sub-groups on definition, approaches and methods Identify priorities of work Trigger joint initiatives

5. Key considerations in IE Methodology: wide range of opinions Demand and Supply for IE: who, what, where Constraints to collective action: costs �delivery over learning� attitude

6. Premises for working together on IE NONIE members have signed up to promote more rigorous impact evaluation. Joint work will be possible only if we take an �eclectic� approach to IE, as recommended in the Paris meeting Equally, agreement on a common definition of the boundaries of IE and some common standards is necessary. Commitment from agencies senior management to push IE forward is required

7. Examples of Joint Working FAO-led multi-stakeholder impact assessment of locust damage in Sahel DFID/IEG impact evaluations Ghana, Primary Education Bangladesh, Nutrition India, Rural Poverty OECD-DAC evaluation of General Budget Support

8. Lessons learnt As in any joint evaluation: specify up-front objectives and expectations share information with all stakeholders ensure ownership by all to facilitate uptake and follow-up of recommendations high transaction costs Rigor in impact evaluation can be reached through different methods

9. Elements of Joint Working Administrative framework Common standards Shared knowledge Capacity building Financing arrangements

10. Three Scenarios �Low Case� �Medium Case� �High Case�

11. (1) Administrative Framework Low Case No secretariat Subset of NONIE members work on specific studies Medium Case Light secretariat which facilitates information sharing and organizes regular meetings to compare notes and push for common standards Work plans not coordinated High Case Full secretariat with dedicated staff, up-to-date website, Help Desk Work plans coordinated between NONIE members�and Opportunities for complementary working with 3IE are exploited

12. (2) Common Standards Low Case No standards endorsed by all NONIE members Medium Case Agreement between NONIE members on minimum criteria for impact evaluation Some cross-NONIE peer review to help embed standards High Case Determined attempt to �push the envelope� on methods Each study subject to quality assurance by another NONIE partner or independent expert

13. (3) Knowledge Sharing Low-Case Sporadic, opportunistic dissemination of approaches, lessons learned Medium-Case Conferences and seminars at regular intervals to share ideas on approaches across NONIE; open to non-NONIE parties (e.g. 3IE) High-Case A NONIE website with links to a common database of studies that is regularly updated, with �gate keeping� to ensure that only studies meeting minimum criteria are included Links to other websites and events on impact evaluation

14. (4) Capacity Building Low-Case Some (mainly �on-the-job�) training of consultants involved in studies Medium-Case Systematic attempts to train individuals in impact evaluation design, targeted at (a) staff in NONIE agencies and (b) counterparts in partner countries High-Case In addition to systematic training, long-term investment in partner country institutions (e.g. policy research institutes) to enhance use of IE and to advocate for it

15. (5) Financing Arrangements Low-Case Limited to a sub-set of NONIE members financing an agreed set of studies Medium-Case Ad hoc arrangements for specific NONIE-wide initiatives No common fund to finance programme costs Admin. cost of �light secretariat� absorbed by host agency High-Case NONIE members put money into a common fund which covers both administrative and programme costs Coordinate with 3IE to finance a series of IEs in complementary areas Possibly�competitive bidding for programme funds

16. How far dare we aim and by when?

17. A Framework to Aim For A framework that supports a continuing debate about what constitutes methodological rigour A framework that produces rigorous studies that are owned by the agencies that sponsor them and the development partners that are involved A framework that builds capacity for conducting and using impact evaluations.