An Introduction to Population-based Survey Experiments

1. An Introduction to Population-based Survey Experiments Diana C. Mutz University of Pennsylvania

2. What are Population-based Survey-experiments? Random sampling + Random assignment = Population-Based Experiments

3. Goal to increase methodological innovation and to expand the diversity of methods used across the social sciences Particularly studies that benefit from combining the internal validity of experiments with the external validity of representative population samples.

4. What is TESS? • NSF-supported infrastructure project that provides social scientists with opportunities for innovative data collection • TESS uses two large-scale data collection instruments • Internet surveys • Telephone Interviews • Our goal is to increase methodological innovation and to expand the diversity of methods used across the social sciences.

5. Experiments over Observational Studies Experiments as the Gold Standard for establishing causality: “There are no complex statistical techniques that provide the power and elegance of a simple experimental design."

6. Representative Population Samples Over Subject Pools • “College sophomores may not be people.” (C. Hovland). • “A greater effort must be made to conduct research on persons from life stages other than late adolescence.” • “… I suspect that even middle-aged people, separated from family and friends and confronted with test-like materials on novel and artificial topics in an academic laboratory, would often behave like college students do.” (D. Sears).

7. Telephone Platform • Allowed extensive pretesting. • Investigators can listen in on pretest interviews, and revise accordingly. • Respondents are sent pre-survey incentives and are paid for their interviews. • Timing can be coordinated with Internet data collection.

8. Internet Platform • Address-based sampling (to include cell-phone-only households) for recruitment. • Free notebook computer and internet access if needed so both internet and non-internet households are included • Payment for study participation • Special subpopulations known in advance • Multimedia: Pictures/Video/Audio • Rapid Turnaround

9. Demise of the Telephone Platform • Decreasing investigator demand for this mode of data collection. • Limitations on experimental stimuli • More difficult to recontact the same respondents (greater attrition) • Evidence on reliability of measures generally better via internet • Ability to randomly sample specialized populations

10. Who is Eligible to Use It? • Any faculty member or graduate student in any social science department anywhere in the world. • Proposals may come from any substantive area in the social sciences as long as they utilize experimental or quasi-experimental designs. • Repeated use is encouraged. Researchers to build on previous findings in subsequent proposals.

11. What kinds of studies are appropriate? • Any experiment or quasi-experimental study designed to explore substantive or methodological hypotheses within the social sciences. • Limits of request for sample and number of questions • Must make a clear contribution to knowledge. • Particularly studies that would benefit from combining the internal validity of experiments with the external validity of representative population samples.

12. How Does It Work? • Proposals are limited to 5 pages in length. • Minimal investment of time in proposal preparation • Encourages fast turnaround from reviewers • Proposals are submitted through an easy online process. • With just a few clicks of the mouse, your proposal enters the review process.

13. Interdisciplinary by Design Diverse Associate PIs oversee the review process… Diversity of topics reduces potential contamination of respondents. Investigators need not share a survey interview.

14. Why do research this way? • Capitalizes on economies of scale to provide more data to more investigators than would otherwise be possible for the same cost. • Encourages social scientists to use methodologies they would not otherwise use. • Encourages quicker pace of research progress • Promotes more policy-relevant research

15. Key Advantages of Population-based Experiments Experimental designs on policy-relevant populations

16. Experimentally investigated what preventative care benefits were most important to consumers Respondents: 500 with a minor present in their home 500 without a minor present in their home Paying More for Prevention? Respondents choose among three health plans, with the possibility of opting into special vaccination coverage.

17. Novel Experimental Treatments, e.g., Threatening Masculinity Respondents randomly assigned to receive threatening or non-threatening feedback on a general knowledge test or a “masculine knowledge test:” • When it is sold in the US, what country does the beer Heineken come from? • What are the true measures of a “two by four”? • Who had back-to-back MVP seasons in the NFL besides Joe Montana? • Which filter on a car typically requires changing every 3000 miles? • What is considered a good maximum benching weight? • Which tool clamps onto an object and will stay clamped without being held in place?

18. Assessing the generalizabilityof previous studies The conventional wisdom about infidelity based on student samples: • Men value sexual fidelity, whereas women value emotional fidelity. The finding when research is based on a representative sample of adults: • Both men and women show more anger and blame over sexual infidelity, and both men and women show greater hurt feelings over emotional infidelity.

19. Race and OCD Inventories Do OCD screening batteries work less well for African-Americans? If so, why? 2 by 2 experiment with representative samples of European-Americans (50%) and African-Americans (50%), asked about ethnic and racial identity either before the OCD battery (to increase salience of race), or after the scales were asked.

20. Race and OCD Inventories 1) Same racial difference in average scores in representative samples as in college student sample. 2) Increasing the salience of race increases African-Americans’ OCD screening scores.

21. Key Advantages of Web-based Research • Experimental designs on policy-relevant populations • Using advance knowledge of respondent characteristics

22. Key Advantages of Web-based Research • Experimental designs on policy-relevant populations • Using advance knowledge of respondent characteristics • Representative sample of participants drawn from the population of individuals whose family income is less than $30,000/year • Representative sample of people residing in coastal areas

23. Using Advance Knowledge for Experimental Treatments:Social Trust and E-Commerce

24. Key Advantages of Web-based Research • Experimental designs on policy-relevant populations • Using advance knowledge of respondent characteristics • More complex, subtle and engaging interactions with respondents

25. Complexity: The Laid-Off Worker Experiment A person is “laid off” because his or her company had to reduce its staff. “Think for a moment about the person and then tell me how much government help, if any, that person should receive while looking for a new job.” One vignette, 96 variations: The person is a [white/black] [male/female] in [his/her] [early 20’s/mid 30’s/early 40’s]. [He/She] is [single/a single parent/married/married and has children] and [is/is not] a dependable worker.

26. Experimental Manipulations

27. Perceptions of Obesity Random samples of Black and White men and women evaluate Black and White, male and female models. Asked to indicate when model becomes “overweight,” and when “obese”

38. Perceptions of Overweight Results: Female image perceived as overweight and obese at significantly smaller body sizes than the male image. Race of image did not matter, but race of perceiver did; White men judged bodies as obese and overweight at smaller body sizes than Black men

39. Seeing is Believing? The Influence of Pictures on Risk Perceptions Does seeing the aftermath of natural disasters affect willingness to insure against them? Tornadoes, Hurricanes, Fires

41. Using words, sound, pictures Disentangling direction of causality: Perceived quality of neighborhood Level of social organization, interaction

43. The Costs of War Do the deaths of private security personnel (i.e., “contract soldiers”) affect Americans less than deaths of military soldiers? Do they elicit less of a response? Do they help insulate support for the war from the death toll?

45. Key Advantages of Web-based Research • Experimental designs on policy-relevant populations • Using advance knowledge of respondent characteristics • More complex, subtle and engaging interactions with respondents • Repeated contact over time

46. Effects of Sotomayor Confirmation Did Sotomayor’s confirmation change Hispanic attitudes toward government? Toward the Supreme Court? Three groups interviewed after nomination, but before confirmation, then again after confirmation

47. Perceptions of Government Fairness, Before and After Sotomayor Confirmation Post-confirmation Nominated, but not confirmed

48. Attitudes Toward Supreme Court, Before and After Sotomayor Confirmation Mean Favorability Toward Supreme Court Post-confirmation Nominated, but not confirmed

49. Key Advantages of Web-based Research • Experimental designs on policy-relevant populations • Using advance knowledge of respondent characteristics • More complex, subtle and engaging interactions with respondents • Repeated contact over time • “Firehouse” studies in response to unfolding events

50. Respondents: Georgia Residents Half of Respondents, African-American Half of Respondents, White Race and attitudes about the confederate flag