2010 Annual Conference Harvard Program in Survey Research October 22, 2010. Survey Experiments: Past, Present, Future. Thomas M. Guterbock Director, Center for Survey Research University of Virginia. Overview. Why survey experiments are so cool Defining the survey experiment
(and what’s not)
Source: Babbie textbook
It’s still a survey experiment even if:
Large, probability-based samples do make the survey experiment better!
This is a questionnaire
But this is not a survey experiment
Source: Babbie textbook
A slippery distinction
. . .This is but a partial picture.
A quick scan of the landscape
The primary corpus of accepted research in survey methods today is almost entirely based on:
(with thanks to the late
Steven L. Nock, my co-author)
Virginia Dept. of Emergency Management
U.S. Dept. of Homeland Security
In-depth survey: average interview length 28 minutes
Fully supported Spanish language interviews as needed
Data collection using CATI (Computer-Assisted Telephone Interviewing)
2,657 interviews conducted by CSR, Sept-Dec 2009.
Triple-frame sample design includes cellphones, landline RDD and listed phones
Inclusion of cellphones increases representativeness
Margin of error: +/- 2.3 percentage points
Weighting by ownership, race, gender, geography, and type of telephone service
Focus: dirty bomb(s) in the NCR
Will residents decide to stay or to go?
3 scenarios at increasing hazard levels: Minimum, moderate, maximum
Respondent is presented with only two of the three tested scenarios
Over 5,000 scenario tests in the study
Four aspects (“factors”) of the scenarios were experimentally varied using random assignment
PATH: Which two hazard levels are asked
NOTICE: Whether the event is preceded by prior notice or threats
LOCATION: The respondent’s location when the event occurs
SOURCE: The source of the information about the event
Notice, location, and source are kept constant for both scenarios asked
The four factors result in 48 different possible versions of the scenario, randomly assigned.
Follow up questions were asked about the decision to shelter in place or evacuate, as appropriate (once only)
Shelter in place detail
Willingness to remain at location, reasons for leaving, what would aid staying
Reason for leaving, destination, mode of travel, needs, use of designated route
Mandatory evacuation: everyone was asked evacuation detail eventually
Percent who perceive “High Risk” or “Very High Risk” (by hazard level)
Will They Stay or Will They Go?
“Based on this information, would you stay at HOME, would you leave immediately to go somewhere else or would you continue with your activities?”
Shelter-in-Place or Evacuation
“Based on this information, would you stay at WORK, would you leave immediately to go somewhere else or would you continue with your activities?”
Shelter-in-Place or Evacuation (cont.)
Location when event occurs: Home
In the minimum scenario, prior notice has a significant effect on the decision to stay or go
Location when event occurs: Work or Other Building
Prior notice has no significant effect
Compliance with shelter in place instruction is highest when the source of information is the State Governor or Mayor of DC
At home, gender effect is significant for all three scenarios.
When event occurs while at work/another building, gender effect is significant in two of three scenarios.
Percentage of people who would leave their home immediately is not large
Many people will leave their place of work if the event is far away (‘minimal hazard’)
Most of these will head to their homes
The scenarios with greater ‘hazard’ did raise perception of risk
But the rates of leaving are similar for moderate and maximum hazards
Higher education, prior positive experience in an emergency, female gender also increase sheltering compliance
Still to come: multivariate analysis using HLM
. . . to the mutual benefit of both!
Steven L. Nock and Thomas M. Guterbock
Chapter in James Wright and Peter Marsden, eds., Handbook of Survey Research, Second Edition. Wiley Interscience, 2010.