Survey experiments
1 / 28

Survey Experiments - PowerPoint PPT Presentation

  • Uploaded on

Survey Experiments. Defined. Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment. Methodological Issues. Missing Data Matching Both can be an issue in experiments other than surveys. Missing Data.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' Survey Experiments' - ivor-cross

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript


  • Uses a survey question as its measurement device

  • Manipulates the content, order, format, or other characteristics of the survey as a treatment

Methodological issues
Methodological Issues

  • Missing Data

  • Matching

  • Both can be an issue in experiments other than surveys

Missing data
Missing Data

  • Some observations missing data on the DV or IVs

  • If missing at random, not a problem to drop from the analysis

  • But usually not missing at random

  • Deleting non-random missing causes bias

Missing data ii
Missing Data II

  • Data can also be missing intentionally:

  • Some cases not “treated”

  • Possible to “guess” what would have happened to a subject had they been in another treatment group

    • Allows within-subject comparison of two treatments, the one they received and the one they could have received

Solution imputation
Solution: Imputation

  • Suppose Yi = a + b1Xi1 + b2Xi2 +ei

  • But Yi missing for some observations

  • Xi1 and Xi2 not missing

  • Regress Y on Xi1 and Xi2 for all non-missing observations

  • Use b1 and b2 to calculate predicted Ypi

Better yet multiple imputation
Better Yet: Multiple Imputation

  • Ypi is a predicted value with uncertainty

  • Multiple imputation predicts multiple values for Ypi drawn from a distribution of predicted values

  • 5 or so predicted Ypi sufficient for inference, no need for many

  • Gary King’s Amelia program available free on-line


  • Experiments can be pre-matched to avoid large random sample

  • Match subjects on important characteristics such as

    • Sex

    • Race

    • Age

    • Education levels

    • Other traits?


  • Often necessary in field experiment when randomization more difficult to control

  • propensity scoreis the probability of an observation being assigned to a particular treatment in a study given a set of known variables.

  • Propensity scores reduce selection bias by equating groups based on these variables


  • Why do people change their answers to survey questions if the order of questions changes?

  • Does changing survey responses indicate that people do not have well-formed opinions


  • Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue

  • Separable Preferences: What a person wants on every issue is independent of what they get on other issues

Measuring nonseparable preferences
Measuring Nonseparable Preferences


  • Randomize the order of pairs of survey questions

    • For some issues, aggregate responses different across question order

  • Each subject answers questions in order

    • Issue 1 then Issue 2

    • Issue 2 then Issue 1


  • Impute what subject would have answered had they heard questions in different order

  • For each question we then have

    Yi (if first) – Yi (if second)

  • One of these will be imputed for each person since they cannot answer a question both first and second in the order

  • First study to analyze individual differences in question orders, not simply aggregate differences


  • Nonseparable preferences explain question order effects

  • Political information level does not

  • Response instability not due to uninformed respondents

Are survey experiments externally valid

Are Survey Experiments Externally Valid?


American Political Science Review 2010


  • Many survey experiments expose subjects to different information to show effect of on responses

  • In a survey experiment, subjects are a “captive audience” that must pay attention

  • Do the same information effects appear in the real world

  • Compare survey experiments with natural experiments


  • Survey experiments give people to political information about immigration and medical care

  • Pre-post survey also in field during change in medical insurance and immigration

    • Ask respondents which media sources they use

  • Is the effect of information in the survey experiment as large as in the natural experiment?