survey experiments
Skip this Video
Download Presentation
Survey Experiments

Loading in 2 Seconds...

play fullscreen
1 / 28

Survey Experiments - PowerPoint PPT Presentation

  • Uploaded on

Survey Experiments. Defined. Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment. Methodological Issues. Missing Data Matching Both can be an issue in experiments other than surveys. Missing Data.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Survey Experiments' - ivor-cross

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
  • Uses a survey question as its measurement device
  • Manipulates the content, order, format, or other characteristics of the survey as a treatment
methodological issues
Methodological Issues
  • Missing Data
  • Matching
  • Both can be an issue in experiments other than surveys
missing data
Missing Data
  • Some observations missing data on the DV or IVs
  • If missing at random, not a problem to drop from the analysis
  • But usually not missing at random
  • Deleting non-random missing causes bias
missing data ii
Missing Data II
  • Data can also be missing intentionally:
  • Some cases not “treated”
  • Possible to “guess” what would have happened to a subject had they been in another treatment group
    • Allows within-subject comparison of two treatments, the one they received and the one they could have received
solution imputation
Solution: Imputation
  • Suppose Yi = a + b1Xi1 + b2Xi2 +ei
  • But Yi missing for some observations
  • Xi1 and Xi2 not missing
  • Regress Y on Xi1 and Xi2 for all non-missing observations
  • Use b1 and b2 to calculate predicted Ypi
better yet multiple imputation
Better Yet: Multiple Imputation
  • Ypi is a predicted value with uncertainty
  • Multiple imputation predicts multiple values for Ypi drawn from a distribution of predicted values
  • 5 or so predicted Ypi sufficient for inference, no need for many
  • Gary King’s Amelia program available free on-line
  • Experiments can be pre-matched to avoid large random sample
  • Match subjects on important characteristics such as
    • Sex
    • Race
    • Age
    • Education levels
    • Other traits?
  • Often necessary in field experiment when randomization more difficult to control
  • propensity scoreis the probability of an observation being assigned to a particular treatment in a study given a set of known variables.
  • Propensity scores reduce selection bias by equating groups based on these variables
  • Why do people change their answers to survey questions if the order of questions changes?
  • Does changing survey responses indicate that people do not have well-formed opinions
  • Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue
  • Separable Preferences: What a person wants on every issue is independent of what they get on other issues
  • Randomize the order of pairs of survey questions
    • For some issues, aggregate responses different across question order
  • Each subject answers questions in order
    • Issue 1 then Issue 2
    • Issue 2 then Issue 1
  • Impute what subject would have answered had they heard questions in different order
  • For each question we then have

Yi (if first) – Yi (if second)

  • One of these will be imputed for each person since they cannot answer a question both first and second in the order
  • First study to analyze individual differences in question orders, not simply aggregate differences
  • Nonseparable preferences explain question order effects
  • Political information level does not
  • Response instability not due to uninformed respondents
are survey experiments externally valid

Are Survey Experiments Externally Valid?


American Political Science Review 2010

  • Many survey experiments expose subjects to different information to show effect of on responses
  • In a survey experiment, subjects are a “captive audience” that must pay attention
  • Do the same information effects appear in the real world
  • Compare survey experiments with natural experiments
  • Survey experiments give people to political information about immigration and medical care
  • Pre-post survey also in field during change in medical insurance and immigration
    • Ask respondents which media sources they use
  • Is the effect of information in the survey experiment as large as in the natural experiment?