Randomized Control Trials for Business Solutions: Putting Them to Work for You

1. Randomized Control Trials for Business Solutions:Putting Them to Work for You Jon Zinman Dartmouth College October 6, 2006

2. Research Team Members • Dean Karlan, Yale University • Sendhil Mullainathan, Harvard University

3. Overview of Talk • Examples of product spaces where academics can help CUs improve profits and member satisfaction (using randomized-control trials) • Focus on mortgages, small business lending • Where $ is! • Biggest opportunity for comparative advantage. Little academic and consultant attention compared to, e.g., saving, financial planning, credit card use 2. Methodology: • How we do it • Operational requirements

4. Example 1: Mortgage Menus • Too much choice depresses consumer demand • Psychology studies • Bertrand-Karlan-Mullainathan-Shafir-Zinman (2005) showed this in an experiment with an actual consumer finance company • But what’s the “right” menu of choices? • “Right” depends on our partner institutions’ objectives, but for now let’s say: • Profitable • Sustainable (long-run member satisfaction)

5. Example 1: Mortgage Menus • Here is a hard choice: • Choice A: 7-1 ARM, no points at 7.5% vs. • Choice B: Fixed rate, no points, at 8%. • However, if you add choice C, • Choice C: Fixed rate, 2 points, at 8.05% • B looks much more attractive and becomes easy choice

6. Outline of a Mortgage Menus Experiment Experiment: 1) Identify small set of promising menus 2) Run horserace in course of actual ops 3) Analyze results and identify best menu(s)

7. Example 2: Refinancing • Evidence suggests that mortgagees procrastinate or don’t pay attention, therefore don’t finance at right time • Business implication. Help members refi at right time and: • Profit (new clients, member retention, more frequent refis) • Increase client satisfaction

8. Outline of a Refi Experiment Marketing for attention. • Small cues, messages drive clients choices • Our prior work shows this. Showing a female photo on direct mail increased consumer loan takeup among dormant male clients by about same amount as a 50% drop in the interest rate • Tap social networks (potential comparative advantage for CUs) • So test a set of promising cues/messages via direct mail, phone

9. Outline of Another Refi Experiment Experiment with automatic refis. • Help members choose refi triggers in advance • comparative advantages here for CUs: trust, personal touch • Offer binding and non-binding commitments • Test takeup rate, profitability, ex-post member satisfaction for a few automatic refi options

10. Example 3:Home equity loan/line utilization • Seems low given: • Simultaneous holdings with relatively expensive credit card debt • # of homeowners who have substantial card debt and no HELOC or HEloan

11. Outline of Home Equity Experiment • Use psychological principles to drive use of equity lines (instead of more expensive forms of borrowing) • Marketing with: • Salient rate comparisons • Cues (picture of family, home improvements) • Make mental accounting work for HELOCs (“put your interest savings back into your house, kids’ education, etc.”)

12. Example 4: Pricing Riskon Small Business Loans • How do you know where to draw the line? • Some lenders leave profitable deals on table. Example from Karlan-Zinman (2006): • Experiment on high-risk consumer loans • Added step to Lender’s normal underwriting process: applicants who were rejected but close to the bar got a loan 25% or 50% of the time (randomly assigned) • These loans ended up being profitable • Also measurably positive impacts on clients • Another method we use for identifying profitability frontier: credit scoring for small businesses

13. What we Mean by “Experiment”:Randomized-Control Trials (RCTs) • What they are: • Identical members or prospective clients get different offers • Offers are randomly assigned (“drawn out of a hat”) within a reasonable, predefined range • range is guided by business considerations

14. Why RCTs? • RCTs are “gold standard” method for identifying causality in social, medical science; biz applications • Highly practical/profitable biz proposition • Our work with smaller firms • Larger firms have integrated RCTs into routine marketing and merchandizing operations, face-to-face contact • Credit card companies • Amazon • Food/drug retailers • H&R Block

15. RCTs: Step-by-Step • Identify product space and key questions • Design experiment to answer key questions • Implement design into operations • Piloting • Full launch • Monitor implementation for compliance with design • Analyze data from experiment • Data sharing: consent and disclosure issues 6. Apply results of experiment

16. RCT Step #1:Identify Area(s) for Experimentation • Process: focused brainstorming • Players: researchers, senior mgmt. • Outcomes: • choice of product spaces • outlines of experimental design like ones I gave earlier

17. RCT Step #2.Design experiment • Process: take design outline, meld it to operational realities of firm. • Players: research team, senior mgmt, field/area managers, front-line personnel, IT personnel • Examples: • Ensure data sharing feasible: consent, disclosures required? • Designing direct mailers • Coordinating experimental direct mailers with routine marketing efforts • Training loan officers to offer a new automatic refi product. Say some clients will get direct mail or face-to-face offer of this new product, other won’t. Then loan officers need to know: • Content of new product(s) • Presentation • Who gets the offer? (compliance with randomization) • Usually involves a software add-on to MIS that loan officer would reference for each client, and automatically prompt a new product offer, or regular product offer • Staffing requirements: • Research team: we bring a project manager • CU: may want dedicated project manager (depends on scale)

18. RCT Step #3:Implement design into operations 3a. Pilot experiment • Small scale • Work out kinks: • Design tweaks • Operational tweaks (compliance, clunkiness) • Fix/eliminate backfires (member complaints, staff burdens) 3b. Full launch

19. RCT Step #4:Monitor Compliance with Design • Audits/site visits for any face-to-face components • Statistical tracking

20. RCT Step #5:Analyze Data from Experiment • Researchers generate a report • Upfront issue: make sure can hand researchers the requisite data needed for analysis • What consent, disclosures are needed, if any?

21. Step #6:Apply Results of Experiment • What have we learned, and what should the CU do next? • Discussions between researchers, mgmt • Lead to recommendations from researchers about how to apply the results: • Specific product, pricing, marketing decisions • Strategy and positioning • Opportunities for additional experiments

22. Take-Aways and Next Steps • RCTs work: standard- and best-practice for answering key strategic questions • Research team has some ideas for high-impact projects • We are looking for CU partners • So let me know if you have interest in RCTs & would like to explore opportunities