Using qualitative and mixed methods in rapid evaluations
This presentation is the property of its rightful owner.
Sponsored Links
1 / 39

Using qualitative and mixed methods in [ rapid] evaluations PowerPoint PPT Presentation


  • 37 Views
  • Uploaded on
  • Presentation posted in: General

Using qualitative and mixed methods in [ rapid] evaluations. Michael Woolcock Lead Social Development Specialist Development Research Group, World Bank Santo Domingo November 14, 2011. Primary source material.

Download Presentation

Using qualitative and mixed methods in [ rapid] evaluations

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Using qualitative and mixed methods in rapid evaluations

Using qualitative and mixed methods in [rapid] evaluations

Michael Woolcock

Lead Social Development Specialist

Development Research Group, World Bank

Santo Domingo

November 14, 2011


Primary source material

Primary source material

  • Bamberger, Michael, VijayendraRao and Michael Woolcock (2010) “Using Mixed Methods in Monitoring and Evaluation: Experiences from International Development”, in Abbas Tashakkori and Charles Teddlie (eds.) Handbook of Mixed Methods (2nd revised edition) Thousand Oaks, CA: Sage Publications, pp. 613-641

  • Barron, Patrick, Rachael Diprose and Michael Woolcock (2011) Contesting Development: Participatory Projects and Local Conflict Dynamics in Indonesia New Haven: Yale University Press

  • Woolcock, Michael (2009) ‘Toward a Plurality of Methods in Project Evaluation: A Contextualized Approach to Understanding Impact Trajectories and Efficacy’ Journal of Development Effectiveness 1(1): 1-14


Overview

Overview

  • What is a ‘rapid assessment’?

  • The limits of rapid assessment

  • Ten reasons to use qualitative methods

  • Three challenges in evaluation:

    • Allocating development resources

    • Assessing project effectiveness (in general)

    • Assessing complex ‘social’ projects (in particular)

  • Discussion of options, strategies for assessing projects using mixed methods

  • Some examples of mixed methods evaluations


What is a rapid assessment

What is a rapid assessment?

  • An evaluation done when time and resources are highly constrained

  • Same principles apply as for comprehensive evaluations…

    • But, if anything, need to be more skilled!

  • Strategy likely to entail deploying a range of methods and tools

    • So need to be conversant across disciplines

  • Setting reasonable expectations is crucial

    • As is anticipating political pressures for certain results


The limits of rapid assessments

The limits of rapid assessments

  • Better than nothing…

  • …but ‘quick and dirty’ risks being just that

  • Difficult to discern nature/extent of impact trajectory

  • Difficult to draw strong causal inference

  • Primacy effects: initial ‘impressions’ endure (perhaps very erroneously)

  • Best as complement/prelude to, not substitute for, comprehensive evaluation


Ten reasons to use qualitative methods in rapid assessments

Ten Reasons to Use Qualitative Methods in (Rapid) Assessments

  • Understanding Political, Social Change

    • ‘Process’ often as important as ‘product’

    • Modernization of rules, social relations, meaning systems

  • Examining Dynamics (not just ‘Demographics’) of Group Membership

    • How are boundaries defined, determined? How are leaders determined?

  • Accessing Sensitive Issues and Stigmatized/Marginalized Groups

    • E.g., conflict and corruption; sex workers


Ten reasons to use qualitative methods in rapid assessments1

Ten Reasons to Use Qualitative Methods in (Rapid) Assessments

  • Explaining Context Idiosyncrasies

    • Beyond “context matters” to understanding how and why, at different units of analysis

    • ‘Contexts’ not merely “out there” but “in here”; the Bank produces legible contexts

  • Unpacking Understandings of Concepts and (‘Fixed’) Categories

    • Surveys assume everyone understands questions and categories the same way; do they?

    • Qualitative methods can be used to correct and/or complement orthodox surveys


Using qualitative and mixed methods in rapid evaluations

Ten Reasons to Use Qualitative Methods in (Rapid) Assessments

  • Facilitating Researcher-Respondent Interaction

    • Enhance two-way flow of information

    • Cross-checking; providing feedback

  • Exploring Alternative Approaches to Understanding ‘Causality’

    • Econometrics: robustness tests on large N datasets; controlling for various contending factors

    • History: single/rare event processes

    • Anthropology: deep knowledge of contexts

    • Exploring inductive approaches

      • Cf. ER doctors, courtroom lawyers, solving jigsaws


Using qualitative and mixed methods in rapid evaluations

Ten Reasons to Use Qualitative Methods in (Rapid) Assessments

  • Observing ‘Unobservables’

    • Project impact not just a function of easily measured factors; unobserved factors—such as motivation, political ties—also important

  • Exploring Characteristics of ‘Outliers’

    • Not necessarily ‘noise’ or ‘exceptional’; can be high instructive (cf. illness informs health)

  • Resolving Apparent Anomalies

    • Nice when inter and intra method results align, but sometimes they don’t; who/which is ‘right’?


Three challenges

Three challenges

  • How to allocate development resources?

  • How to assess project effectiveness in general?

  • How to assess complex social development projects in particular?


Allocating development resources

Allocating development resources

  • How to allocate finite resources to projects believed likely to have a positive development impact?

  • Allocations made for good and bad reasons, only a part of which is ‘evidence-based’…

    • … but most of which is ‘theory-based’, i.e., done because of an implicit (if not explicit) belief that Intervention A will ‘cause’ Impact B in Place C net of Factors D and E for Reasons F and G

      • E.g., micro-credit will raise the income of villagers in Flores, independently of their education and wealth, because it enhances their capacity to respond to shocks (floods, illness) and enables larger-scale investment in productive assets (seeds, fertilizer)


Allocating development resources1

Allocating development resources

  • Imperatives of large development agencies strongly favor one-size-fits-all policy solutions (despite protestations to the contrary!). An ideal project yields…

    • predictable, readily-measurable, quick, photogenic, non-controversial, context-independent results

      • roads, electrification, immunization

  • Want ‘best practices’ that ‘work’ that can be readily scaled up and replicated

  • Projects that diverge from this structure enter the resource allocation game at a distinct disadvantage…

    • … but the obligation to demonstrate impact (rightly) remains; just need to enter the fray well armed

      • empirically, theoretically, strategically


Key principles

Key principles

  • Ask interesting and important questions, then assemble the best combination of methods to answer it

    • Not, “What questions can I answer with this data?”

    • Not, “I don’t have a randomized design, so therefore I can’t say anything defensible”

  • Generate data to help projects ‘learn’, in real time

    • Be useful, here and now

    • Make ‘M’ as cool as ‘E’

  • Help to more carefully identify the conditions under which given interventions ‘work’

    • Individual methods, per se, are not inherently ‘rigorous’; they become so to the extent they appropriately match the problems they confront, the constraints they overcome

    • Focus on understanding SD as much as determining LATE


How to assess project effectiveness

How to Assess Project Effectiveness?

  • Need to disentangle the effect of a given intervention over and above other factors occurring simultaneously

    • Distinguishing between the ‘signal’ and ‘noise’

      • Is my job creation program reducing unemployment, or is it just the booming economy?

  • An intervention itself may have many components

    • TTLs are most immediately concerned about which aspect is the most important, or the binding constraint

    • (Important as this is, it is not the same thing as assessing impact)

  • Need to be able to make defensible causal claims about project efficacy even (especially) when the apparent ‘rigor’ of econometric methods aren’t suitable/available

    • Thus need to change both the terms and content of debate


Making knowledge claims in project evaluation and development research

Making knowledge claims in project evaluation and development research

  • Construct validity

    • How well does my instrument assess the underlying concepts (‘poverty’, ‘participation’, ‘conflict’, ‘empowerment’)?

  • Internal validity

    • How well have I addressed various sources of bias (most notably selection effects) influencing the relationship between IV and DV?

      • i.e., what is my identification strategy?

  • External validity

    • How well can I extrapolate my findings? If my project works ‘here’, will it also work ‘there’? If it works with ‘them’, will it work with ‘these’? Will bigger be better?


We observe an outcome indicator

We observe an outcome indicator…

Intervention


And its value rises after the program

…and its value rises after the program

Intervention


However we need to identify the counterfactual i e what would have happened otherwise

However, we need to identify the counterfactual (i.e., what would have happened otherwise)…

Intervention


Since only then can we determine the impact of the intervention

… since only then can we determine the impact of the intervention


Why are complex interventions so hard to evaluate a simple example

Why are ‘complex’ interventions so hard to evaluate? A simple example

  • You are the inventor of ‘BrightSmile’, a new toothpaste that you are sure makes teeth whiter and reduces cavities without any harmful side effects. How would you ‘prove’ this to public health officials and (say) Colgate?


Why are complex interventions so hard to evaluate a simple example1

Why are ‘complex’ interventions so hard to evaluate? A simple example

  • You are the inventor of ‘BrightSmile’, a new toothpaste that you are sure makes teeth whiter and reduces cavities without any harmful side effects. How would you ‘prove’ this to public health officials and (say) Colgate?

  • Hopefully (!), you would be able to:

    • Randomly assign participants to a ‘treatment’ and ‘control’ group (and then have then switch after a certain period); make sure both groups brushed the same way, with the same frequency, using the same amount of paste and the same type of brush; ensure nobody (except an administrator, who did not do the data analysis) knew who was in which group


Demonstrating impact of brightsmile vs sd projects

Demonstrating ‘impact’ of BrightSmile vs. SD projects

  • Enormously difficult—methodologically, logistically and empirically—to formally identify ‘impact’; equally problematic to draw general ‘policy implications’, especially for other countries

  • Prototypical “complex” CDD/J4P project:

    • Open project menu: unconstrained content of intervention

    • Highly participatory: communities control resources and decision-making

    • Decentralized: local providers and communities given high degree of discretion in implementation

    • Emphasis on building capabilities and the capacity for collective action

    • Context-specific; project is (in principle) designed to respond to and reflect local cultural realities

    • Project’s impact may be ‘non-additive’ (e.g., stepwise, exponential, high initially then tapering off…)


Complexity and evaluation

‘Complexity’ and Evaluation

Low

Many

Narrow

Wide


How does j4p work over time or what is its functional form

How does J4P work over time?(or, what is its ‘functional form’?)

‘Governance’?

CCTs?

Impact

Impact

A

B

Time

Time

Bridges?

‘AIDS awareness’?

Impact

Impact

C

D

Time

Time


How does j4p work over time or what is its functional form1

How does J4P work over time?(or, what is its ‘functional form’?)

Unintended consequences?

Shocks?

(‘Impulse response

function’)

Impact

Impact

E

F

Time

Time

Land titling?

‘Empowerment’?

Impact

Impact

G

H

Time

Time


How does j4p work over time or what is its functional form2

How does J4P work over time?(or, what is its ‘functional form’?)

?

Impact

Impact

Unknown… Unknowable?

J

I

Time

Time


So what can we do when

So, what can we do when…

  • Inputs are variables (not constants)?

    • Facilitation/participation vs. tax cuts (seeds, pills, etc)

    • Teaching vs. text books

    • Therapy vs. medicine

  • Adapting to context is an explicit, desirable feature?

    • Each context/project nexus is thus idiosyncratic

  • Time, resources are very limited?

  • Outcomes are inherently hard to define and measure?

    • E.g., empowerment, collective action, conflict mediation, social capital


Use mixed methods

Use mixed methods

  • Combinations of methods to complement strengths and weaknesses of each

  • Understanding context, process

  • Enhancing construct, internal and external validity

    • Scaling up, replication

  • Especially as it pertains to making causal claims

    • Econometrics vs history vs anthropology vs law

  • Link to explicit theory of change


Other uses for mixed methods

Other uses for Mixed Methods

  • When existing time and resources prelude doing or using formal survey/census data

    • Examples: St Lucia and Colombia

  • When it’s unclear what “intervention” might be responsible for observed outcomes

    • That is, no clear ex ante hypotheses; working inductively from matched comparison cases

      • Examples:

        • Putnam (1993) on regional governance in Italy

        • Mahoney (2010) on governance in Central America

        • Collins (2001) on “good to great” US companies

        • Varshney (2002) on sources of ethnic violence in India


  • Practical examples 1

    1. Poverty in Guatemala (GUAPA)

    ‘Parallel’

    Quan: expanded LSMS

    first social capital module

    large differences by region, gender, income, ethnicity

    pervasive elite capture

    Qual: 10 villages (5 different ethnic groups)

    perceptions of exclusion, access to services

    fear of reprisal, of children being stolen

    legacy of shocks (political and natural)

    links to LSMS data

    Practical examples (1)


    Practical examples 2

    2. Poverty in Delhi slums (Jha, Rao and Woolcock 2007)

    ‘Sequential’

    Qual: 4 migrant communities

    near, far, recent, long-term

    Quan: 800 randomly selected representative households

    From survival to mobility

    role of norms (sharing, status) and networks (kinship, politics)

    housing, employment transitions

    property rights

    Understanding ‘governance’

    managing collective action

    crucial role of service provision

    Practical examples (2)


    Practical examples 3

    Practical examples (3)

    3. ‘Justice for the Poor’ Initiative

    • Origins in Indonesia

      • Draws on the approach and findings from large local conflict study

    • Integrated qualitative and quantitative approach

    • Results show importance of understanding

      • Rules of the game (meta-rules)

      • Dynamics of difference (politics of ‘us’-‘them’ relations)

      • Efficacy of intermediaries (legitimacy, enforceability)

  • Extension to Cambodia…

    • Research on collective disputes (e.g., land), to inform IDA grant in 2007

  • …and now into Africa and East Asia

    • Sierra Leone, Kenya, Vanuatu, East Timor, PNG…


  • J4p core research design

    J4P: Core Research Design

    • Enormous investment in recruiting, training, keeping local field staff

    • Training centers on techniques, ethics, data management and analysis

    • Where possible, use existing quantitative data sources to (a) complement qualitative work, and (b) help with sampling

    • Sampling based on basic comparative method:

      • Maximum difference between contexts

      • Focus on outliers (‘exceptions to the rule’)

    • Rough rule of thumb: analysis takes three times as long as data collection

      • Analysis can’t be “outsourced”: research team needs to be involved at all stages


    Practical examples 4

    Practical examples (4)

    • Assessing the Kechamatan Development Program (KDP, now PNPM) in Indonesia, the world’s largest CDD program


    Using qualitative and mixed methods in rapid evaluations

    Contesting Development

    Participatory Projects and Local

    Conflict Dynamics in Indonesia

    PATRICK BARRON

    RACHAEL DIPROSE

    MICHAEL WOOLCOCK

    Yale University Press, 2011


    Summary of methods used in barron diprose and woolcock 2011

    Summary of methods used in Barron, Diprose and Woolcock (2011)

    Breadth

    Breadth

    PODES,

    GDS

    Depth

    Newspaper

    Analysis

    Depth

    Key Informant Survey

    Case Studies


    Summary of findings

    Summary of findings


    Implications for policy and practice

    Implications for policy and practice

    • Non-linear trajectories of change

      • J-curves, step-functions, ?? (not a straight line)

    • Social relations

      • As resources, as constraints

      • Virtues and limits of traditional dispute resolution

    • Capacity to engage

      • Facilitators as street-level diplomats (“Habermasian bureaucrats”)

      • Low capacity a donor problem as much as a client ‘gap’

    • State-society relations, “good governance” a product of

      • Good struggles

        • ‘Interim institutions’ forged, legitimated through equitable contests

      • Good failures

        • Learning organizations as platforms of innovation, feedback, accountability

        • Tolerating, rewarding lots of experiments (many won’t work)


    Concluding thoughts

    The virtues and limits of measurement

    Tension between simplifying versus complicating reality

    Triangulation

    Integrating more data, better data, more diverse data as “substitutes” and “complements”

    Surveys as tool for adaptation and guidance

    Not prescription for uniformity or control

    One size (literally) does not fit all

    Encouraging comparability across time and space

    Concluding thoughts


  • Login