survey experiments
Download
Skip this Video
Download Presentation
Survey Experiments

Loading in 2 Seconds...

play fullscreen
1 / 28

Survey Experiments - PowerPoint PPT Presentation


  • 66 Views
  • Uploaded on

Survey Experiments. Defined. Uses a survey question as its measurement device Manipulates the content, order, format, or other characteristics of the survey as a treatment. Methodological Issues. Missing Data Matching Both can be an issue in experiments other than surveys. Missing Data.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Survey Experiments' - ivor-cross


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
defined
Defined
  • Uses a survey question as its measurement device
  • Manipulates the content, order, format, or other characteristics of the survey as a treatment
methodological issues
Methodological Issues
  • Missing Data
  • Matching
  • Both can be an issue in experiments other than surveys
missing data
Missing Data
  • Some observations missing data on the DV or IVs
  • If missing at random, not a problem to drop from the analysis
  • But usually not missing at random
  • Deleting non-random missing causes bias
missing data ii
Missing Data II
  • Data can also be missing intentionally:
  • Some cases not “treated”
  • Possible to “guess” what would have happened to a subject had they been in another treatment group
    • Allows within-subject comparison of two treatments, the one they received and the one they could have received
solution imputation
Solution: Imputation
  • Suppose Yi = a + b1Xi1 + b2Xi2 +ei
  • But Yi missing for some observations
  • Xi1 and Xi2 not missing
  • Regress Y on Xi1 and Xi2 for all non-missing observations
  • Use b1 and b2 to calculate predicted Ypi
better yet multiple imputation
Better Yet: Multiple Imputation
  • Ypi is a predicted value with uncertainty
  • Multiple imputation predicts multiple values for Ypi drawn from a distribution of predicted values
  • 5 or so predicted Ypi sufficient for inference, no need for many
  • Gary King’s Amelia program available free on-line
matching
Matching
  • Experiments can be pre-matched to avoid large random sample
  • Match subjects on important characteristics such as
    • Sex
    • Race
    • Age
    • Education levels
    • Other traits?
matching1
Matching
  • Often necessary in field experiment when randomization more difficult to control
  • propensity scoreis the probability of an observation being assigned to a particular treatment in a study given a set of known variables.
  • Propensity scores reduce selection bias by equating groups based on these variables
question
Question
  • Why do people change their answers to survey questions if the order of questions changes?
  • Does changing survey responses indicate that people do not have well-formed opinions
theory
Theory
  • Nonseparable Preferences: What a person wants on one issue depends on what she gets on another issue
  • Separable Preferences: What a person wants on every issue is independent of what they get on other issues
method
Method
  • Randomize the order of pairs of survey questions
    • For some issues, aggregate responses different across question order
  • Each subject answers questions in order
    • Issue 1 then Issue 2
    • Issue 2 then Issue 1
method1
Method
  • Impute what subject would have answered had they heard questions in different order
  • For each question we then have

Yi (if first) – Yi (if second)

  • One of these will be imputed for each person since they cannot answer a question both first and second in the order
  • First study to analyze individual differences in question orders, not simply aggregate differences
conclusions
Conclusions
  • Nonseparable preferences explain question order effects
  • Political information level does not
  • Response instability not due to uninformed respondents
are survey experiments externally valid

Are Survey Experiments Externally Valid?

JASON BARABAS and JENNIFER JERIT

American Political Science Review 2010

question1
Question
  • Many survey experiments expose subjects to different information to show effect of on responses
  • In a survey experiment, subjects are a “captive audience” that must pay attention
  • Do the same information effects appear in the real world
  • Compare survey experiments with natural experiments
method2
Method
  • Survey experiments give people to political information about immigration and medical care
  • Pre-post survey also in field during change in medical insurance and immigration
    • Ask respondents which media sources they use
  • Is the effect of information in the survey experiment as large as in the natural experiment?
ad