Man versus machine

1. Dan KlermanComment on Glaeser, Hillis, Kim, Kominers & LucaHow Does Restaurant Compliance Affect the Returns to Algorithms?Evidence from Boston’s Restaurant Inspectors NBER Law & Economics Summer InstituteJuly 25, 2019

2. Man versus machine

3. Cyborg Chess

4. What is the right question? • Man v machine? • Human v algorithm? • Human v cyborg v machine • Human alone v algorithm-assisted human v pure algorithm • What question is answered by paper? • Framing of paper • Human v algorithm • Algorithm better • So need to improve human compliance with algorithms • Klerman view • Paper may only show that algorithm-assisted humans are better than humans alone • May need new experiment or new analysis to show that algorithms better than humans • If algorithm-assisted humans do best • Need ways to help humans and computers work better together

5. Paper isHuman v Algorithm-Assisted Human • Inspectors given a list (docket) of approximately 45 restaurants • 15 ranked highest in advance by inspector (“business as usual”) • 15 ranked highest by data-rich algorithm • 15 ranked highest by data-poor algorithm • Inspector then chose which restaurants to inspect • Did not inspect all • Had discretion about which to inspect • So only have outcome (inspection) data on restaurants chosen by inspectors • Of those restaurants, 50% more violations when restaurant was ranked high by algorithm • BUT inspectors may be using private information to choose high-violation restaurants • If inspectors had inspected all restaurants ranked highly by algorithms, maybe # of violations would have been lower

6. Potential Author Response • Private information is already incorporated in rankings in advance by inspector • But • inspector is likely to spend much more time evaluating restaurants on docket that will actually have to spend time inspecting • Than when asked to rank all restaurants in his area • Also human judgment may be better at excluding low-violation restaurants than choosing high-violation restaurants

7. How to Test Human v Algorithm • New experiment • Same procedure, except inspectors don’t have discretion • Must inspect restaurants in order presented in docket • Or must inspected at least first 10 restaurants on docket • Reanalysis of data already have • Need to understand how inspectors chose restaurants to inspect • Why didn’t just choose restaurants that inspector had previously ranked high? • Paper seems to assume that inspectors chose restaurants randomly • So restaurants inspected are representative sample of those on the docket • Possible, but unlikely

8. How Did Inspectors Choose Restaurants to Inspect? • Did inspectors choose algorithmically-listed restaurants if geographically very close to inspector-listed restaurants? • If so, then quasi-random and analysis may shed light on human v algorithm question • At least if restricted to algorithmically listed restaurants close to inspector-listed restaurants • Did inspectors choose algorithmically-listed restaurants if listed higher on docket or more often? • If so, then quasi-random and analysis may shed light on human v algorithm question • At least if restricted to algorithmically-listed restaurants listed high on docket • Is there heterogeneity among inspectors that could be exploited? • Did some just go down the list? • Did inspectors choose algorithmically-docketed restaurants based on private information? • Compare to prior inspector ranking • Then paper cannot shed light on human v algorithm question

9. Test for Whether Algorithm-Assisted Humans Do Better • Some restaurants put on docket (list) because ranked highly both by • Inspector in advance • 1 or both algorithms • Do restaurants put on docket (list) by both human and algorithm have more violations than those put on docket only by algorithm? • Instead of: • And creating two observations for restaurants inspected and on docket because ranked high by two methods • Try • If and other coefficients for restaurants ranked highly by both humans and algorithms do better than restaurants ranked highly only by algorithms • Suggests that algorithm-assisted humans can do better than algorithms alone

10. The Future • Is human v algorithm the right question? • Maybe cyborgs (algorithm-assisted humans) can do best? • That is a question worth investigating • And not just be accident • Could design experiment in which inspectors sometimes given discretion and sometimes not • Hoffman & Kahn (QJE 2017) suggests human discretion makes decisionmaking worse • Hiring decisions better if use algorithm without human discretion • Chess • Humans assisted by computers beat computers …

11. The Future • Is human v algorithm the right question? • Maybe cyborgs (algorithm-assisted humans) can do best? • That is a question worth investigating • And not just be accident • Could design experiment in which inspectors sometimes given discretion and sometimes not • Hoffman & Kahn (QJE 2017) suggests human discretion makes decisionmaking worse • Hiring decisions better if use algorithm without human discretion • Chess • Humans assisted by computers beat computers until 2017