Randomized Experimentation for the Program Manager: A Quick How-To Guide

1. Randomized Experimentationfor the Program Manager:A Quick How-To Guide Jonathan Zinman Assistant Professor of Economics, Dartmouth College Research Associate, Innovations for Poverty Action May 1, 2007 Presentation for IFC M&E Conference

2. What’s Your Objective? • If it’s “beat or meet the market”…. • Can’t afford to focus on evaluation per se

3. What’s Your Strategy? • Focus must be…. Using evaluation to feed innovation • (Abhijit’s “dynamic learning”)

4. Experiment Evaluate Innovate Experimentation &the Learning Organization:A Virtuous Cycle

5. Randomized Evaluation:A Quick How-to Guide • What should I evaluate? • How do I design and implement the evaluation? • How do I interpret and apply the results? At each step will highlight how randomized-control trials (RCTs) can be part of an innovation strategy producing comparative advantage for: • Client financial institutions (FIs) • Wholesale investors like IFC

6. What Should I Evaluate? • Which interventions (“treatments”)? • Which outcomes?

7. Which Interventions? • Use RCTs to address critical questions: • Business/programmatic/policy strategy • Stakes should be high, given fixed costs of experimentation and related data collection • Existing programs you’re funding (of course!) • Also….

8. Which Interventions? • But also: “Microfinancial Engineering” • Analogy to “high” finance • “Design-build” partnerships between academics & financial institutions • Input from funders, program managers, policymakers • Don’t just “take opportunities” to do systematic experimentation • Create opportunities. What’s a “program”? • Innovation challenges • Brokering • Funding

9. Which Interventions? • RCT interventions can be built into any critical business function • Product development • Pricing • Marketing and targeting • Monitoring and enforcement • Risk assessment

10. Which Interventions? Product Development examples • Savings products with new features: • Commitment • Goals • Method and nature of communication with client • Smoking cessation performance bond • Smokers “bet themselves” they can quit (as measured by urine test in 6 months) • Financial institution fills missing market: enforcement • Micro-insurance Evaluate by randomizing • Whether product offered • Product features

11. Which Interventions? Pricing examples • Loans: commercial, consumer • Savings: various products • Micro-insurance Evaluate by randomizing • Initial offer • Dynamic pricing (future offers)

12. Which Interventions? Marketing and targeting examples • Marketing: what speaks to the client and why? • Direct mail: randomize content • Text messaging: randomize content • Targeting: finding your market or intended beneficiaries • Health insurance for the poor in the Philippines • Working with government, its contractors to experiment with: • Measurement techniques • Incentives for using them • Compensation • Punishment (monitoring, auditing)

13. Which Interventions? Loan monitoring/enforcement examples: • Require peer monitoring or liability? • Incentivize peer monitoring (referrals)? • How monitor and communicating with (delinquent) borrowers? • What are the most (cost-) effective threats and penalties? Evaluate by randomizing: • Mechanisms, incentives, protocols

14. Which Interventions? Risk assessment • Possible to lend profitably to some rejected applicants? • Innovations to test: • Improve risk assessment model (credit scoring; other projects) • Provide incentives to overcome loan officer conservatism (details to follow) Evaluate by randomizing: • The approve/reject decision (within reasonable bounds)

15. How-to Example From One Project • Experiment with risk assessment • Objective: measure impacts of expanding access to consumer loans • Popular product in South Africa • Worked with a leading for-profit “microlender” • “Traditional” micro(enterprise)credit largely absent in South Africa • 4-month installment loans; 200% APR

16. Example: Expanding Access What we did. Overview of the experiment: • Intervention: randomly approve loans for some close-to-creditworthy (“marginal”) applicants who would normally be rejected • This is the “treatment group” • Other marginal applicants remain rejected • This is the “control group” • Measure impacts • Difference between treatment and control • For the many outcomes of interest • Lender: use its data to calculate profits • Applicants: survey data on economic and well-being outcomes • See “Expanding Access: Using Randomized Supply Decisions to Estimate the Impacts” (with Dean Karlan): http://www.dartmouth.edu/~jzinman/Papers/Derationing.Karlan-Zinman.pdf

17. How Did ThisEvaluation Come About? • Remember Step 1: What do we evaluate? • Which intervention(s)? • Which outcomes? • This project very much design-build • Lender motivation • Prior experiments with Lender • Identified specific market failures (asymmetric information problems) • Identified binding liquidity constraints for borrowers • These reinforced Lender’s priors that loan officers being too conservative • Profitable deals being left on the table • Open to RCT as systematic way to control and evaluate risk of liberalizing criteria

18. Motivation for this Evaluation:What do we Evaluate? Researcher/policy/funder angles: • Consumer credit controversial • Policy tends to restrict rather than encourage access • But why? Economic arguments for restricting tenuous • But consumer (micro)credit markets growing • Our methodology applicable to microenterprise credit as well

19. Step 2. How do we do it?Design and Implementation 3 key issues in this case: • Scope of study • Implementing the intervention: how to randomize loan approvals decisions • Tracking and measuring outcomes

20. Design & Implementation: Scope A. Scope: How big? Where? • Required deal flow for a conclusive evaluation? • How big a sample do we need to answer the questions of interest (statistical power) • Researchers identify • Best way to obtain the required deal flow? Researchers and Lender worked together to identify: • A practical definition of “marginal” applicant • Timeframe for the intervention (2 months) • Participating branches. Chose 8 branches that would • Produce required deal flow in the 2 month timeframe • Be relatively easy to monitor and train • Be representative enough to draw conclusions re: whether or not to scale up the intervention to other branches

21. Design & Implementation:The Intervention B. How actually randomize loan approvals? • Insert research protocols into loan assessment process. In this case 2 additional steps: • Loan officers rejected applicants into “marginal” and “egregious” • New software randomizes “marginal” into “keep rejected” or “approve” (second look) • New implementations streamline this with introduction of pure credit scoring model • Train branch personnel • Monitor (& incentivize) branch personnel to comply with protocols

22. Design & Implementation: Measurement C. Tracking & measuring outcomes • Lender data on sales, costs, & hence profitability • Follow-up survey data on applicant outcomes: • Economic (employment status, income, consumption) • Subjective well-being (decision power, optimism, mental health) • Researchers designed household survey instrument • Contract survey administration to survey firm • Close monitoring from pilot to final survey

23. Results: Lender Outcomes • Lender made money: marginal loans were profitable • Less profitable than loans above the bar • But profitable nonetheless • Even on initial loan • Profits from acquiring new clients even bigger • Did Lender scale up? That was the plan, but…. • Then Lender was merged into a larger bank • New senior mgmt hostile to “consultants” • Old senior mgmt (our partners) banked knowledge and took to new firms

24. Results: Applicant Outcomes Large, positive, statistically significant impacts on: • Economic self-sufficiency (employment, income, above poverty line) • Consumption (avoiding hunger, food quality) • Outlook and control (decision power, optimism) No significant impacts on: • Investment (education, housing, self-employment) • Physical health Negative impact (90% significant) on mental health (depression, stress) Overall impact significant and positive • If weight all outcomes equally

25. Step 3. How Apply The Results? The intervention itself • Do (social) costs exceed benefits? • In this case interpreting results simple: win-win • Often there are tradeoffs: weighing costs and benefits requires some insight into the “whys” of impacts • Here evidence of market failures from earlier experiments • Prior and project evidence of binding liquidity constraints • Opportunity cost of intervention(s)? • In consumer credit key is ruling out negative effects: default policy/programmatic approach is to restrict access • Unlike microenterprise credit, where default approach is to expand/subsidize access, and hence opportunity cost of subsidy matters

26. How Apply The Results? Applying the results (external validity) • Scalability • Replicability Three complementary approaches: • Design so that get answers re: why interventions do or don’t work • Choose sites/markets/partners carefully • Do lots of RCT experimentation

27. Take-Aways RCTs deliver: • Gold-standard measures of impacts • Insights into the “why” questions that: • Affect scalability • Feed back into innovation RCTs are doable: • Design-build partnerships with researchers for: • Microfinancial Engineering • Innovation that is scalable and replicable

28. Experiment Evaluate Innovate Experimentation &the Learning Organization:A Virtuous Cycle