slide1 n.
Download
Skip this Video
Download Presentation
Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM

Loading in 2 Seconds...

play fullscreen
1 / 28

Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM - PowerPoint PPT Presentation


  • 104 Views
  • Uploaded on

Experiments Exploring the Relationship (or Lack Thereof) Between Nonresponse Error and Measurement Error. Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM Chris Toppe, Points of Light Foundation Courtney Kennedy, MPSM Ting Yan, NORC. Acknowledgement.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Roger Tourangeau, SRC, JPSM Robert M. Groves, SRC, JPSM Stanley Presser, JPSM' - tanner-farley


Download Now An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Experiments Exploring the Relationship (or Lack Thereof) Between Nonresponse Error and Measurement Error

Roger Tourangeau, SRC, JPSM

Robert M. Groves, SRC, JPSM

Stanley Presser, JPSM

Chris Toppe, Points of Light Foundation

Courtney Kennedy, MPSM

Ting Yan, NORC

acknowledgement
Acknowledgement
  • The work reported here was supported by a grant from the National Science Foundation (SES 550385) to Roger Tourangeau and Robert Groves. We are grateful to NSF for its support and especially to Cheryl Eavey.
  • The National Science Foundation is not responsible for the conclusions presented here.
  • We thank Duston Pope for his contributions to the design, John LaFrance, and Mirta Galesic for their contributions to the studies discussed here study design.

Nonresponse Error and Measurement Error

outline of talk
Outline of Talk
  • Models of the relationship between nonresponse and measurement error
  • Experiment 1: A study of volunteers and volunteering
  • Experiment 2: Topic and sponsorship in a two phase web surveys
  • Two additional studies
    • Gain-loss framing and response propensities
    • An experiment with voters and nonvoters
  • Conclusions

Nonresponse Error and Measurement Error

models of the nonresponse measurement error relationship
Models of the Nonresponse-Measurement Error Relationship
  • Underlying model

Nonresponse Error and Measurement Error

model 1 independence model
Model 1: Independence Model
  • Nonresponse essentially motivational (interest in topic, altruism, low opportunity costs)
  • Error largely cognitive (poor comprehensive of the items, memory difficulties, poor estimation strategies)

M

C

Up

p

e

Ue

model 2 reluctance
Model 2: Reluctance
  • Difficult to persuade respondents answer sloppily (Bollinger and David, 2001; Fricker 2006)—Common cause model

M

p

Up

e

Ue

Nonresponse Error and Measurement Error

model 3 topic sensitivity
Model 3: Topic Sensitivity
  • Drug users and reports about drug use

X

p

e

Up

Ue

Nonresponse Error and Measurement Error

model 4 topic sponsor effects
Model 4: Topic, Sponsor Effects
  • Presentation of the survey request affects both who responds and how they respond

Topic Interest

Topic

p

e

Up

Ue

Nonresponse Error and Measurement Error

framing effects
“Framing” Effects
  • Why should the description of the topic or sponsor effect the answers Rs give?
  • Three mechanisms
    • Construal: Description of the survey may affect how they interpret the questions (National Crime Survey vs. National Survey of Violence Against Women—the latter may suggest a broader interpretation of “sexual assault” than the former)
    • Priming: Topic, sponsor may serve as a retrieval cue, making some memories more accessible
    • Perception of what is relevant or useful—Norenzayan and Schwarz (1999): Institute for Social Research and Institute for Personality Research
  • Evidence for all three (Galesic and Tourangeau, 2007)

Nonresponse Error and Measurement Error

experiment 1 the march of dimes study
Experiment 1: The March of Dimes Study
  • Survey topics
    • Survey of Giving and Volunteering
    • Survey of Jobs and Labor Force Activity
  • Target populations
    • March of Dimes mailing list
      • Donors, volunteers, other nominated by donors
    • General adult sample from vendor list
  • Sponsor
    • March of Dimes
    • University of Michigan

Nonresponse Error and Measurement Error

recruitment protocol
Recruitment Protocol
  • Mailed paper questionnaire
  • Advance letter
  • Initial questionnaire packet
  • Reminder postcard
  • Followup questionnaire packet

All materials emphasized sponsor and topic

Nonresponse Error and Measurement Error

10

response rate by topic and population
Response Rate by Topic and Population

Concern about sugging?

Nonresponse Error and Measurement Error

11

mixed results on commitment and poor match to records
Mixed Results on Commitment and Poor Match to Records

Nonresponse Error and Measurement Error

13

conclusions from preliminary analysis
Conclusions from Preliminary Analysis

Volunteers and donors to March of Dimes tend to participate in all survey topics/sponsor combinations

Those MoD donors with greater commitment more positive to the MoD sponorship than those with lesser commitment

Mixed results on whether those with greater commitment tend to misreport their activities in support of MoD

Nonresponse Error and Measurement Error

14

experiment 2 two wave web survey
Experiment 2: Two-Wave Web Survey
  • Collected data from members of two web panels in on-line survey
  • Wave 1 collected predictors of response propensities and answers to second wave
  • Wave 1 respondents (n=3000) invited to take part in second survey about a month later
  • Ostensible topic and sponsorship of second survey varied
  • Examined how response rates varied as a function of attitudes toward topic and sponsor
  • Also looked at effects on answers

Nonresponse Error and Measurement Error

experiment 2 first wave
Experiment 2: First Wave
  • n=47,078 asked to complete University of Michigan survey on “attitudes and lifestyles”
  • Members of two web panels: SSI (4.1% responded), e-rewards (14.8%); 3,000 respondents in all
  • Wave 1 items included:
    • Questions on how often the respondent had responded to web survey requests in the past;
    • Items assessing the respondents’ views about the importance of a number of social issues, including issues related to the topic of the second questionnaire (crime and violence, gun control);
    • Items assessing support for or opposition to gun control policies;
    • Relevant factual items (such as the presence of a gun in the household and experiences using guns);
    • Rating scales on the prestige of various organizations, including the purported sponsors of the second

Nonresponse Error and Measurement Error

experiment 2 second wave
Experiment 2: Second Wave
  • n=2,020 (67.3 response rate conditional on wave 1)
  • Varied three factors—sponsor, topic, and purported length
    • Sponsor:
      • National Coalition of Gun Owners (pro-Gun)
      • National Coalition for Victims of Gun Violence (anti-gun)
      • National Center for the Study of Crime
    • Topic
      • Crime prevention
      • Gun control policies
    • Length (20 minutes, 5 minutes, no mention)

Nonresponse Error and Measurement Error

our hypotheses
Our Hypotheses
  • Attitudes toward the sponsor and topic to affect both the likelihood of cooperation and response behavior on questions relevant to the sponsors’ likely point of view.
  • Similarly, we expected topic interest to affect response propensities and answers to questions in the survey
  • Expected burden to lower response rates

Nonresponse Error and Measurement Error

results response rates
Results—Response Rates

The experimental variables seem to have little impact on either the response rates or nonresponse bias in estimates in the second survey.

results answers
Results — Answers
  • Created three composites
    • Personal safety (4 items) — no effects
    • Gun control attitudes (11 items)
      • Main effect for topic: More support for gun control when the topic was described as crime prevention than when it was described as gun control—context effect?
      • Purported sponsor interacted with wave 1 att’s (next slide)
    • Involvement in issue of gun control (3 items)—topic effect among pro-gun types: those below the first survey median in their support for gun control measures reported greater involvement in the issue when the topic was described as crime prevention (mean of 2.47) than when it was described as gun control (mean of 2.36)

Nonresponse Error and Measurement Error

results liking for the sponsor and sloppiness
Results — Liking for the Sponsor and Sloppiness

Nonresponse Error and Measurement Error

two new studies gain loss framing
Two New Studies: Gain-Loss Framing
  • Gain-Loss Framing: Two-wave telephone study; request to do second interview at end of first interview

We really appreciate the help you’ve given us today. We are interested in how people’s views about the issues we discussed today change over time so it is important we talk to you again in a couple of weeks.

Then get either

The information you’ve already provided to us will be a lot more valuable if you complete the second interview (77.9%)

Or

Unfortunately, the information you’ve already provided to us will be much less valuable unless you complete the second interview (87.5%)

Nonresponse Error and Measurement Error

voting study
Voting Study
  • Obtained sample of Maryland voters and non-voters
  • Vary mode of data collection (mail vs. telephone), incentive ($5 vs. none), and topic (Health and Lifestyles vs. Politics, Elections, and Voting)
  • Attempt to test “topic sensitivity” model (Model 3)
  • Reasons to think that dosage could be higher or lower in mail

Nonresponse Error and Measurement Error

conclusions
Conclusions
  • Every study is framed somehow
  • Framing can effect response rates, though not necessarily nonresponse error
    • Topic effects on response rates found in Groves, Presser, and Dipko (2004), Experiments 1 and 2 here, “Birding study” in Groves et al. (2006)
    • But not always—three other studies in Groves et al. (2006) don’t find topic effects
    • Sponsor can effect responses, but again not always
  • Framing can effect answers, though effects may be subtle as in Experiment 2 here and in earlier experiment by Galesic and Tourangeau (2007)
  • Only weak evidence that reluctant R’s, or those who dislike sponsor or topic more prone to measurement error

Nonresponse Error and Measurement Error