Humanitarian aid evaluation at Medecins sans Frontieres

1. Humanitarian aid evaluation at Medecins sans Frontieres Sabine Kampmüller, MIH MSF Vienna Evaluation Unit http://evaluation.msf.at

2. Dimensions of evaluation: Source: SDC (2002)

3. Evaluation post-disaster: Haiti earthquake response Evaluation conducted 5 – 10/2010

4. Evaluation process • 1st time (!) all section Review • 6 specific reviews: Global/operational, medical/surg, log/supply, Comms, FR, HR • Quantitative & qualitative methods • data, field visits, interviews, qualitative research - beneficiaries, web survey -staff • Limitations: • Incomplete and non-uniform data • Recall problems

6. Patients / Community’s Perception • Very positive perception of MSF´s interventions and services • Fear/experience of stigmatisation, increase of violence in the camps  • Lack of space for occupational and educational activities.   • More attention to socio-cultural and spiritual needs

7. Haiti evaluation findings: • MSF was one of the biggest emergency health actors • Operational choices in line with emergency needs; less coherent over time • Common data collection difficult to impossible • International platforms / working groups are underused / undervalued • Successful adocacy on some issues, complicated decision making prevented more

8. Main recommendations: • Organize inter-section capacity for assessment and monitoring of evolving needs and assistance • Define strategy on mass casualty • Revise emergency supply strategies • Ensure uniform data collection • Strengthen technical working groups • Focus on organisational learning • Engage with national and international actors

9. Thematic Evaluation: Response to displacement Evaluation conducted 2009/2010

10. Evaluation process • Comparison of 6 case studies (urban/rural, low/middle income, etc) • Quantitative & qualitative methods • Literature research, document/tools & data review, field visits, interviews, • Limitations: • Availability of data and key people • Poor documentation

11. Particularity of open settings • Lack of clear boundaries : • Geographical spread, invisibility, needs difficult to identify and measure, protection issues • Displaced settle in environment with available resources and existing health system • Better survival capacities, but deterioration likely • Protracted, chronic or intermittent character: • Mortality near-normal levels (pre-emergency), might rise slowly over time

12. Findings on assessment • Complexity of open settings requires more attention to assessment • Critical aspects of information missing • Quantitative information difficult to obtain • Health system issues, access barriers little addressed • Concerns on the use of surveys • Views of displaced and host often omitted • Capacities and vulnerabilities not assessed

13. Recommendations on assessment • Develop innovative assessment approaches for inaccessible areas • Distance assessment“ • Promote systematic use of qualitative methods • Adopt concept of “continual” assessment • Develop a frame to assess vulnerabilities, capacities and coping • Provide better support and guidance • assessment toolbox, experts, training

14. Findings on intervention Engagement with the health system a main challenge Outreach workers invaluable, set ups improvable Non-medical assistance is marginal Overambitious targets of coverage Strategy adapted to the level of emergency

15. Recommendations on intervention • Need for new intervention frameworks • Adopt existing models • Generalize the practice to cover both displaced and host where appropriate • Define the criteria / scope of “light support” • Develop community based strategies

16. Evaluation criteria (donors) Relevance Effectiveness Efficiency Impact Sustainability Appropriateness (Connectedness) (Coherence) Timeliness Coordination Coverage Adopted from: Hallam, A. Good practice review (ODI), 1998