combining different methods in impact evaluation l.
Skip this Video
Loading SlideShow in 5 Seconds..
Combining different methods in impact evaluation PowerPoint Presentation
Download Presentation
Combining different methods in impact evaluation

Loading in 2 Seconds...

play fullscreen
1 / 17

Combining different methods in impact evaluation - PowerPoint PPT Presentation

  • Uploaded on

Combining different methods in impact evaluation . Presentation by Steen Folke, Senior researcher at the Danish Institute for International Studies International Workshop on Impact Evaluation Paris, November 15 2006. Important banalities.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Combining different methods in impact evaluation' - Faraday

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
combining different methods in impact evaluation
Combining different methods in impact evaluation

Presentation by Steen Folke,

Senior researcher at the Danish Institute for

International Studies

International Workshop on Impact Evaluation

Paris, November 15 2006

important banalities
Important banalities
  • Impact evaluation is not an exact science – and it is dangerous to pretend that it is
  • The use of a combination of different – quantitative and qualitative – methods can highlight different dimensions of impact
  • Impact is the result of development processes that depend on the context as well as the intervention
definition of impact
Definition of impact
  • ”Impact assessment is the systematic analysis of the lasting or significant changes – positive or negative, intended or not – in people’s lives brought about by a given action or series of actions”

(Chris Roche: Impact Assessment for Development Agencies, Oxford 1999)

classical effect evaluations
Classical effect evaluations
  • Quasi-experimental survey design
  • Ceteris paribus assumptions
  • Before/after and/or with/without
  • Quantitative methods
  • Attempted objectivity
  • Dubious assumptions
participatory impact assessments
Participatory impact assessments
  • Involvement of beneficiaries
  • External facilitators
  • Participatory techniques
  • Qualitative methods
  • Subjective
  • Problems of reliability
wider impact studies
Wider impact studies
  • Heterogeneous category of in-depth studies
  • Contextualised, tailor-made approach
  • Unintended consequences as well as stated objectives
  • Extensive fieldwork
  • Combination of quantitative and qualitative methods
  • Development interventions and societal processes
noakhali rural development project bangladesh
Noakhali Rural Development Project, Bangladesh
  • Danida funding 1978-92 390 m. DKK (> $50 m.) – flagship in Danish aid
  • 2 phases, 15 components: irrigation, cooperatives, rural poor prog., mass education etc.
  • >60 expatriate advisers, staff >1000
  • Waning enthusiasm, many implementation problems (complexity)
  • 3rd phase planned, but aborted
ex post impact evaluation
Ex-post impact evaluation
  • Did the flagship float or sink?
  • No terminal evaluation
  • Ex-post impact study 9 years after
  • Contextualised, tailor-made approach
  • Combination of quantitative and qualitative methods
  • 8 researchers, 15 assistants, 4 months fieldwork
methods i project focus
Methods I: Project focus
  • Documentary study (project documents)
  • Archival work in Danish embassy, Dhaka
  • Questionnaire survey with former advisers and Danida staff
  • Stakeholder interviews (Danida staff, former advisers, Bangladeshi staff)
  • Quantitative analysis of project monitoring data
methods ii study of context
Methods II: Study of context
  • The national context: books, articles, statistics (economy, policies)
  • The local context: census data, other sources (environment, population, socio-economic development)
  • Institutional mapping (esp. NGOs)
  • Extensive village studies (12 villages, 9 with and 3 without NRDP)
  • Intensive village studies (4 NRDP villages, restudy 20 years later)
methods iii quantitative
Methods III: Quantitative
  • Surveys of 5 important project components: irrigation, infrastructure, fisheries, cooperatives for rural poor, mass education
  • Random sampling (beneficiaries)
  • Questionnaire-based interviews with beneficiaries
  • Some interviews with non-beneficiaries (’control group’)
methods iv qualitative
Methods IV: Qualitative
  • Assessment of roads, buildings and irrigation canals (function, maintenance)
  • Key informant interviews
  • Focus group interviews
  • Observation
  • In-depth interviews (issue-based and life stories)
example of findings irrigation ii
Example of findings: Irrigation II

Most important reason for improvement in

economic situation compared to 15 years

ago (total: 59 respondents):

Increased production (irrigation): 20 resp.

Remittances: 20 resp.

Petty trade/business: 8 resp.

Other reasons: 11 resp.

Most important reasons for deterioration (17 resp):

Illness, large family, loss in business

conclusion the project
Conclusion: The project
  • The Danida flagship did not sink – the evaluation team found substantial impact in many areas 9 years after
  • The impact was primarily in the form of marginal improvements for the rural population, mainly the poor and women
  • The socio-economic inequalities and the local power structure were not challenged
conclusion the methods
Conclusion: The methods
  • The combination of a range of different methods contributed to unravel different dimensions of impact from a complex project
  • Study of the context was essential to understand impact of the project versus other factors
  • Quantitative and qualitative methods were complementary
  • Attempts to assess the counterfactual brought limited insight