Operational Aspects of Impact Evaluation Arianna Legovini Impact Evaluation Cluster, AFTRL
Operational guidance: Key steps 3. Prepare data collection instruments 4. Collect baseline 1. Identify policy questions 2. Design evaluation 5. Implement program in treatment areas 6. Collect follow up 8. Improve program and implement new 7. Analyze and feedback
Operational guidance:Keep it useful! • “Interesting” is what the government wants • Questioning program helps strengthen design • Provide answers to questions early • Change program as a result • Get better results for client and Bank!
Keep it relevant! 1. Right questions 2. Right team 3. Develop client capacity
Manage trade-offsMethod &Data • Experimental • Least costly • Best estimates • Not always politically feasible • Non-experimental • More expensive • Good second best estimates • Always feasible • Administrative data (expanded to control) • Most cost effective • Piggy back on ongoing data collection • Cost effective • Coordination intensive • Collect special surveys • Tailored and timely • Expensive
Budget • Data collection • Consultants • Staff time • Include baseline costs in PPF • Include follow up costs in project costs • Use Trust Funds to finance consultants • Initiate product IE-code and obtain BB supervision funds Financing
M&E Operational messages • Plan M&E while planning project • Build an explicit team • Develop budget and financing plan • Influence roll-out to obtain control groups • Use quantitative & public allocation criteria • Randomization is ethical • Strengthen monitoring systems