Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops

Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops Rohini Pande Mohammad Kamal Professor of Public Policy 12 August, 2013 Evidence for Policy Design Initiative @CID, Harvard University

The NRLM Framework • Proposal: • Increase household income of rural poor via sustainable livelihood enhancement and improved access to financial and non-financial services • Mission components: • Social, financial, and economic inclusion • Mission features: • Decentralized implementation • Prospective design of M&E framework

NRLM M&E: What we want to know? • Why do we expect NRLM to achieve its goals? • How do we know that, in practice, it is achieving these goals? • If it isn’t, how can we modify it so that it does achieve the goals?

Talk Outline • A Holistic view of M&E: Smart Policy Design • Policy development • Monitoring and Feedback loops • Working with smart policy design and feedback loops • Testing design features for microfinance • MIS systems as feedback mechanisms: Data visualization in MNREGA • Final summing up

Smart Policy Design: Policy Development What is the policy goal What is your theory for how to achieve the goal? Is existing evidence consistent with the theory? Refine objectives, theory Policy design

Smart Policy Design: Monitoring and Feedback Loops Policy design and pilot rollout monitor Is policy achieving its goals Refine objectives, theory Large scale rollout but keep monitoring

Smart Policy Design: Testing Design Features in Microfinance • Objective: • Promotion and growth of micro-enterprises

From Objective to Policy • Objective: • Support growth of micro-enterprises. • Hypothesis: • Micro-enterprises have high growth potential but are limited by lack of financing. • Policy: Access to Microfinance

Smart Policy Design: Policy Development What is the policy goal What is your theory for how to achieve the goal? Is existing evidence consistent with the theory? Refine objectives, theory Policy design

Is existing evidence consistent with theory? • The typical evidence provided to examine microfinance impacts: • High returns to providing enterprises cash • 4-5% monthly returns • High rate of growth of microfinance use through India • High repayment rates • With most MFIs posting over 90% repayment rates Concludes: Microloans are a sustainable way of providing the poor finance

Is existing evidence consistent with hypothesis? • But is that the correct evidence support the hypothesis? • Directly test impact of microfinance on enterprize growth • Rigorous impact evaluations show small average impact Triangulate evidence to reformulate design • Why could this be given that micro-entrepreneurs have high growth potential and site credit constraints as a predominant obstacle to the growth of their businesses? • How can we design an evaluation to test where the answer lies? For redesign: Pilot new policies

How should policy be redesigned?or what feedback is needed? • One possibility: • Replace private sector providers by community organizations • But why will this solve the problem? • Another possibility (and one we test) • Maybe the microfinance contract is too rigid and this implies entrepreneurs don’t undertake high risk high return activities.

From Objective to Policy Goal Policy goal : Encourage entrepreneurship Theory: Entrepreneurs limited by lack of credit Empirics: High returns to capital, but impact of microfinance limited Conceptualframework Refine policy: What limits returns to microfinance? Policy design

Smart Policy Design: Monitoring and Feedback Loops What is the pilot policy How will we monitor? Does the evidence enable better policy design?

Identify pilot policy • What do we know about microenterprises? • Subject to frequent household and demand shocks • High rate of business closings • Lack of insurance to overcome shocks • Irregular income streams with businesses highly subject to seasonality • Conclude: • Microentrepreneurs unable to make profitable investments that require a longer return horizon • Need flexibility in loan contract to make such investments

What do we know about microfinance contracts? • Standard Grameen microcredit loan contract • Immediate repayment obligation: One week between loan disbursement and first repayment • Frequent loan repayment: Weekly repayment • Rigid contract • Conclude: • Test impact of increasing repayment flexibility to microentrepreneurs. • Provide a grace period between loan disbursal and repayment

Recall need for evaluation • Competing theories! • 1. Our theory: Presented above • 2. Competing theory: Contractual flexibility can have adverse effects

Smart Policy Design Method What is the pilot policy? How to monitor

Isevidence consistent with the theory? • The question: • Does a grace period before loan repayment starts enable enterprise growth? • Method to test the question1: • Work with large MFI provider in Kolkata • By lottery select some MFI groups that receivethe grace period contract • Other groups remain on standard contract 1 Field, Pande, Papp, and Rigol (2013)

Pilot design • Half of study sample receives standard contract • Immediate repayment obligation:Two weeks between loan disbursement and first repayment • Loan repayment:Fortnightly repayment • Half of study sample receives contract with a grace period • Immediate repayment obligation:Two months between loan disbursement and first repayment • Loan repayment: Fortnightly repayment

Results

Smart Policy Design: Monitoring and Feedback Loops Policy design and pilot rollout monitor Is policy achieving its goals Refine objectives, theory Large scale rollout but keep monitoring

Using this evidence to enable smart policy design for NRLM • Under NRLMs SHGschoose loan repayment terms for group members. • IS Trade-offs faced by SHG similar to MFI providers? • If SHG has incentives to ensure high repayment but the group doesn’t benefit from single member’s success then they may go for conservative rigid repayment schedule • Policy question: How can SHG incentives be aligned with those of individual member? • What is the hypothesis? • What evaluation mechanism exists for feedback?

MIS and Smart Policy Design: Example of NREGA • Smart policy design is about building feedback loops into the policy process. • The NREGA data visualization project seeks to do this by: • Identifying key program performance indicators • Presenting descriptive statistics with an eye toward digestibility and aesthetics • Facilitating basic hypothesis testing – investigate correlations between key outcomes and administrative inputs • Making program data easier to access for the public and academics (encourage external feedback)

Smart Policy Design Framework:First causal hypotheses • Use simple descriptive statistics to identify problem areas and confirm/reject initial hypotheses • Develop and investigate explanations for observed phenomena • Start with simple correlations between policy inputs and outcomes; move on to more complex regression analysis, field evaluations; update hypotheses at each iteration • Idea to build increasingly defensible causal explanations

NREGA implementation: Need for focus • The NREGA MIS collects dozens of indicators at multiple levels of granularity • Even highly motivated implementers may not have capacity to sift through information and get a clear picture of program performance • These need to be distilled into a few key measures of program performance such as female and SC/ST participation or average time to payment.

Example: female participation in NREGA • Getting data on female participation currently takes 4+ clicks (if you know where to look). • District-level data only available state-wise, block-level data only available district wise – hard to get a broad view of performance without resorting to data scraping • Increasing data accessibility is low-hanging fruit

Reusable charts can increase data accessibility • Web technology exists to pull data from MIS and display it as interactive graphics • Charts are reusable – we can use the same chart format to display many user-selected indicators from the MIS • Technology (JavaScript and the d3.js library) is open source with a large support community

Reusable charts can increase data accessibility

Ongoing work: Data visualization as a feedback loop • How should administrators use MIS/data visualization to identify potential policy levers? • Possible hypotheses: • Demand Side: Absolute supply of work affects female participation through male-first work allocation • Supply side: Higher female empowerment implies greater female participation • First step (work in progress!): Develop software that combines different data to generate user-defined scatter • Such initial diagnostics can help kickstart smart policy design.

Demand side: Examining correlations

Supply side: Examining correlations

Smart Policy Design: The process • Identify your objective • Develop a theory that explains why a specific policy will achieve the objective • What is the mechanism by which the policy causesthe objective to be achieved? • What is evidence that these causal links will work? • How similar is problem to other contexts from which we have evidence on similar policies? • What more do you need to know? • Use monitoring and feedback loops to generate this evidence

Takeaways for Designing NRLM M&E • M&E should not stand apart from policy design and implementation process. • Rather it’s design should ensure feedback loops with policy process. • Shouldhavelittle to no distinction between how to think about monitoring versus evaluation. • Monitoring mechanisms should enable concurrent evaluation. • In all cases, iterate between theory, existing evidence and pilots.

Issues and Challenges in M&E of Rural Livelihoods Programs: Designing Feedback Loops