Field Experiments in Political Science: A Brief History and Some Illustrative Examples

1. Field Experiments in Political Science: A Brief History and Some Illustrative Examples Don Green Columbia University

2. What is an experiment? • Units of analysis are randomly assigned with known probability to treatment and control conditions • Distinction between random and haphazard assignment • Distinction between random assignment and random sampling • In the physical sciences, perfectly controlled experiments substitute for randomized trials

3. What is a field experiment? • Random assignment takes place in a naturalistic setting, enhancing generalizability • Ideally, experimenters play an unobtrusive role, reducing the risk of a violation of symmetry between treatment and control • Four dimensions of naturalism: subjects, treatments, contexts, outcome measures

4. Early (near random?) field experiments in Political Science • Harold Gosnell: Mobilizing voters in the 1924 and 1925 elections • George Hartmann: Using rational or emotional appeals to increase the socialist vote in Allentown, PA in the 1935 elections • Underhill Moore and Charles Callahan: examined the effects of varying New Haven’s parking regulations, traffic controls, and police enforcement in 1930s

5. Early Field Experiments using Random Assignment • Hovland, Lumdsdaine, and Sheffield: Propaganda studies conducted for Experimental Section of the Research Division of the War Department • Samuel Eldersveld: Partisan and nonpartisan voter mobilization campaigns in the 1953 and 1954 Ann Arbor, MI elections • Ironic that randomized field experiments should die out just as they are gaining prominence in other disciplines (e.g., polio vaccine trials)

6. Dominant Modes of Behavioral Research in Political Science • Surveys: spearheaded in the 1950s by American National Election Studies using random sampling • Econometric analysis of aggregate data over time and/or space: growing computing power and technical facility of the discipline during 1970s • Fundamental trade off between research design and post hoc statistical correctives

7. Illusion of Observational Learning Theorem (Gerber, Green, and Kaplan 2004) • When confronted with mixture of observational and experimental evidence, Bayes’ Rule says assign zero weight to observational evidence unless you have informative priors about its bias • When confronted with laboratory experimental evidence, assign it zero weight unless you have informative priors about the biases associated with your extrapolation to some population/setting of interest

8. Implications of the “Illusion” for scientific practice • Researchers using observational data are oblivious to the fact that they routinely underreport the true degree of statistical uncertainty associated with their findings • Misallocation of the discipline’s research portfolio: No field experiments conducted between 1985 and 1998 • Misconceptions about meta-analysis

9. Reclaiming the Experimental Tradition in Social Science • Reassessment of evidence for a variety of basic behavioral propositions by subjecting them to the same level of scrutiny as pharmaceutical evaluations • Secondary aim is to stimulate experimental reflection even in domains of political science where experimentation is infeasible • What follows is a brief overview of selected projects, some of which refute the “it can’t be done” critique

10. Project 1: Voter Mobilization(Authors: Gerber and Green) • 1998 New Haven Study, N=31,098 • Interventions: nonpartisan face-to-face canvassing, commercial phone banks, direct mail • Voter turnout measured using public records • Subsequently replicated with hundreds of thousands of observations in a variety of sites

11. Synthesis of recent randomized experiments on voter turnout: door-to-door canvassing, leafleting, phone calls, direct mail, and e-mail

12. Forest Plot of 85 direct mail experiments – excluding “social pressure“StudiesATE = 0.109 ppts(-0.07,0.290)

13. Lessons Learned • Face-to-Face canvassing raises turnout by approximately 7-9 percentage-points • Volunteer phone banks have moderate effects (3-5 percentage-points) • commercial phone banks are typically ineffective, as are robo-calls • Direct mail has weak effects (except as noted below) • E-mail has no apparent effect

14. Project 2: Habit Formation(Authors: Gerber, Green, and Shachar) • Infer habit from long-term effects of randomized intervention • Example of “downstream experimentation” • Follow up study of people in the 1998 New Haven Study showed that treatment group voted at higher rates • Subsequently replicated in 4 of 5 studies • For each 100 additional votes generated in this election, an additional 33 votes are generated in the next election

15. Project 3: Interpersonal Influence(Author: Nickerson) • Inference without problems of unobserved heterogeneity, which plague other influence studies • Placebo-control design: 2 voter households receive either get-out-the-vote message or a recycling appeal • Also, a control group gets nothing (as expected, turnout is significantly higher in GOTV vs. control)

16. Project 3: Interpersonal Influence(Author: Nickerson) • Findings show that housemates of registered voters who were contacted in the treatment group were significantly more likely to vote than housemates of those who were contacted in the placebo group

17. Example: Nickerson’s influence experiment in Denver 2002

18. Project(s) 4: Influence of Television and Radio • Cable system experiments in 2003, 2004: influence of GOTV ads on turnout • Radio ads in 2005 and 2006 municipal elections: influence on competitiveness • Broadcast TV, cable TV, and radio in the context of a $2 million gubernatorial campaign • Influence of radio on ethnic reconciliation in Rwanda • Spanish language radio and voter turnout

19. Project 5: Social Pressure and Voter Turnout (Gerber, Green, and Larimer) • Using field experiments to test basic behavioral theories • Longstanding interest in “prescriptive” norms dating back to Gosnell’s work in the 1920s • To what extent can one manipulate the salience of “extrinsic” incentives associated with voting?

20. Study Design: August 2006 • Sample: 180,002 households in Michigan, registered voters who voted in 2004 • Setting: August primary election, “open” but contested only on the Republican side • Assignment: 10,000 clusters of 18 households each; in each cluster, households assigned at random to one of five groups: Control, Civic Duty, Hawthorne, Self, and Neighbors • Outcome: Voting in the primary election, as indicated by official records for each individual

21. For more information: (517) 351-1975 email: etov@grebner.com Practical Political Consulting P. O. Box 6249 East Lansing, MI 48826 ECRLOT **C050 THE WAYNE FAMILY 9999 OAK ST FLINT MI 48507 Dear Registered Voter: WHO VOTES IS PUBLIC INFORMATION! Why do so many people fail to vote? We've been talking about the problem for years, but it only seems to get worse. This year, we're taking a different approach. We are reminding people that who votes is a matter of public record. The chart shows your name from the list of registered voters, showing past votes, as well as an empty box which we will fill in to show whether you vote in the August 8 primary election. We intend to mail you an updated chart when we have that information. We will leave the box blank if you do not vote. DO YOUR CIVIC DUTY - VOTE! ----------------------------------------------------------- OAK ST Aug 04 Nov 04 Aug 06 9999 ROBERT SMITH Voted ______ 9999 LAURA BETH Voted Voted ______ 21

22. For more information: (517) 351-1975 email: etov@grebner.com Practical Political Consulting P. O. Box 6249 East Lansing, MI 48826 ECRLOT **C050 THE JACKSON FAMILY 9999 MAPLE DR FLINT MI 48507 Dear Registered Voter: WHAT IF YOUR NEIGHBORS KNEW WHETHER YOU VOTED? Why do so many people fail to vote? We've been talking about the problem for years, but it only seems to get worse. This year, we're taking a new approach. We're sending this mailing to you and your neighbors to publicize who does and does not vote. The chart shows the names of some of your neighbors, showing which have voted in the past. After the August 8 election, we intend to mail an updated chart. You and your neighbors will all know who voted and who did not. DO YOUR CIVIC DUTY - VOTE! ----------------------------------------------------------- MAPLE DR Aug 04 Nov 04 Aug 06 9995 JOSEPH JAMES SMITH Voted Voted ______ 9995 JENNIFER KAY SMITH Voted ______ 9997 RICHARD B JACKSON Voted ______ 9999 KATHY MARIE JACKSON Voted ______ 9999 BRIAN JOSEPH JACKSON Voted ______ 9991 JENNIFER KAY THOMPSON Voted ______ 9991 BOB R THOMPSON Voted ______ 9993 BILL S SMITH ______ 9989 WILLIAM LUKE CASPER Voted ______ 9989 JENNIFER SUE CASPER Voted ______ 9987 MARIA S JOHNSON Voted Voted ______ 9987 TOM JACK JOHNSON Voted Voted ______ 22

24. Project 6: The Effects of Criminal Sentences on Recidivism (Green and Winik) • Random assignment of judges creates analytic leverage • ~1000 defendants in Washington, DC drug courts randomly assigned to nine “calendars” with different sentencing proclivities • No effect of incarceration or length of sentence on recidivism

25. Miscellaneous Randomized Experiments of Interest in the Domain of Political Attiudes and Actions • Randomly varying rules about legislative seniority, term length, and floor recognition • Discrimination experiments focusing on the responsiveness of legislators to constituents of varying ethnic, racial, or partisan profile • Effects of audits and “accountability” interventions • Exposure to draft, school, or visa lotteries and their effects on attitudes and behavior

26. Bottom Line • Randomized experimentation in real world settings give social scientists access to the kinds of practical knowledge that outsiders care about • Possibility, of course, for premature extrapolation of experimental results: e.g., class size experiments • Importance of creating firm empirical foundation for theoretical development and policy intervention (e.g., prejudice reduction)

27. For a recent (2012) discussion of social science experiments and experimental design, see…