1 / 11

Adapting to Real-World Constraints

Adapting to Real-World Constraints. Managing Common Challenges in the Evaluation Process. Overview. Background GEM-3 Project SI Evaluation Selected Challenges Number of Evaluation Questions Contingency Planning Reconciling conflicting data Audience examples and Q&A.

aiko
Download Presentation

Adapting to Real-World Constraints

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Adapting to Real-World Constraints Managing Common Challenges in the Evaluation Process Social Impact

  2. Overview • Background • GEM-3 Project • SI Evaluation • Selected Challenges • Number of Evaluation Questions • Contingency Planning • Reconciling conflicting data • Audience examples and Q&A Social Impact

  3. USAID/Philippines Growth with Equity in Mindanao III Program (GEM-3) • 5-year (2008 to 2012), $99m program in the ARMM and other conflict-affected areas. • Largest and most diverse program (2/3 Mission funding). • Followed $22.3m, GEM-1 (1995-2002) and $82m GEM-2 (2002-2007) programs. • Despite 18 years, $203m+ USG funding, no external evaluation had been conducted. • GEM-3 had three overarching objectives: • Accelerate economic growth; • Ensure that as many people as possible participate in and benefit from that growth; and • Bring about and consolidate peace in Mindanao. • Seven components: (1) Infrastructure; (2) Workforce; (3) Governance; (4) Business growth; (5) Former combatant reintegration; (6) Communications and public relations; and (7) Logistical and perational support. Social Impact

  4. Background on Evaluation • In June 2012, Social Impact (SI)contracted to conduct a final evaluation to examine successes, shortcomings, and lessons learned. • Mixed-methods design involving desk review; key informant and group interviews; household survey; a cost-efficiency/ERR; and site observation. • 11-member evaluation team and over six weeks of fieldwork in Manila and Mindanao. • Video used for evaluation trainings. Social Impact

  5. Evaluation Questions: How many is too many? • Common challenge for both commissioners and evaluators. • Need sufficient breadth and depth. • Finite resources and time. • Balance what is possible and what can be actually be answered. • Ultimately, the questions chosen will dictate the design and direction of evaluation will follow. • Given high visibility and large number of stakeholders, 54 questions were included in SOW. Social Impact

  6. Tips For Refining Questions • Aim for 3-7 primary questions. • Prioritize and rank. • Questions should be realistic and answerable. • Link to the evaluation purpose. • Choose questions which reflect real stakeholders’ needs and interests. • Data simulation exercise. • Don’t use questions which contain two or more questions in one. • Avoid simple yes/no questions. Social Impact

  7. Contingency Planning: Hope for the best, Plan for the worst • “The best laid schemes of mice and men/Often go awry” ~ R. Burns • Challenges on GEM-3 Evaluation: • Weather. • Interviewees asleep. • Logistical challenges. • Site inaccessibility due to conflict. • Asking sensitive questions. Social Impact

  8. Tips for Planning • Plan for known challenges (weather conditions, holidays, school closing, harvest, etc.). • Prepare contingencies. • Alternate sites. • Remote data collection. • Use of local evaluators. • Secondary sources. Social Impact

  9. The Truth, The Whole Truth, and Nothing But…. • Challenges on GEM-3 Evaluation: • Different answers given to different evaluators. • Military escorts. • Presence of cameraman. • Pleasing the interviewers. Social Impact

  10. Tips for Reconciling Conflict Data • “The truth is rarely pure and never simple.” ~ Oscar Wilde • Introduction and Confidentiality • Sequencing • Varied phrasing • Use of local evaluators • Triangulation • Randomized Response Technique Social Impact

  11. What about your experiences? • What are some of your experiences/solutions? • What other common challenges have you found? Social Impact

More Related