1 / 21

How Much of Interviewer Variance is Really Nonresponse Error Variance ?

How Much of Interviewer Variance is Really Nonresponse Error Variance ?. Brady T. West Michigan Program in Survey Methodology University of Michigan-Ann Arbor Kristen Olson Survey Research and Methodology Program University of Nebraska-Lincoln June 14, 2010

Download Presentation

How Much of Interviewer Variance is Really Nonresponse Error Variance ?

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. How Much of Interviewer Varianceis Really Nonresponse Error Variance? Brady T. West Michigan Program in Survey Methodology University of Michigan-Ann Arbor Kristen Olson Survey Research and Methodology Program University of Nebraska-Lincoln June 14, 2010 International Total Survey Error Workshop 2010 1

  2. Interviewer Variance: The Problem • An undesirable product of the data collection process, given interpenetrated sample designs • Responses for same interviewer are more similar than responses for different interviewers • Leads to inflation of variance in survey estimates due to intra-interviewer correlation, ρint • ρint = 0.01, 30 cases per interviewer  13.6% increase in SE of estimates • ρint usually less than 0.02, but can be larger 2

  3. Research Question • Does ρint arise from complex interviewer-respondent interactions / probing for hard items? • High estimates of ρint (0.03–0.12) for factual (easy) and self-completion items in literature… • There is also consistent empirical evidence of interviewer variance in response rates • One estimates ρint with respondent data only, ignoring contributions of NR error variance • How much of interviewer variance can be attributed to nonresponse error variance? 3

  4. The Wisconsin Divorce Study (WDS) • SRS of divorce records from four Wisconsin counties in 1989 and 1993 • Sampled divorce records included official information also collected in a survey • The present study focuses on data collected using CATI: interviewer effects are likely attenuated relative to CAPI • n = 733 cases randomly sampled, and 355 CATI interviews performed by 31 trained interviewers 4

  5. WDS Data • Six Survey Variables of Interest • Length of Marriage in Months • Time since Divorce in Months • Time since Marriage in Months • Number of Marriages including the Divorce • Age at Marriage • Age at Divorce • Date of Divorce was recorded by an official body; other frame measures were reported by one member of the couple (possible errors) 5

  6. Assigning Nonrespondents • Ideally, cases would not be worked by multiple interviewers (e.g., Singer and Frankel, 1982) • WDS used refusal conversion, and there were frequent changes in interviewers working non-finalized cases • This complicates the process of assigning nonrespondents to interviewers 6

  7. Assigning Nonrespondents • Assumption: Interviewers working a particular shift work a random subsample of cases • The focus of this study is on interviewer variance within a shift, rather than across shifts • Persons with different characteristics are likely to be contacted at different times of the day • Account for shift: avoids possible confounding of nonresponse error with differences across shifts 7

  8. Assigning Nonrespondents • Definitions of shifts: • Weekday, 9-5pm (Shift 1: 26.8% of calls) • Weekday, after 5pm (Shift 2: 44.1% of calls) • Weekend, any time (Shift 3: 29.1% of calls) • Similar to work of Stokes and Yeh (1988) • Interviewers worked multiple shifts • Alternative shifts were also considered, and the study results did not change 8

  9. Assigning Nonrespondents • Respondents were assigned to the interviewer completing the interview • Contacted refusals were assigned to: • the first interviewer receiving a refusal, or • the last interviewer to make contact • Non-contacts were assigned to the last interviewer making a call to the case • Random assignment of non-contacts was also considered; no change in results 9

  10. Assigning Nonrespondents • Limited power: 19-24 interviewers worked each shift, based on assignments • Large variability in assigned workloads across interviewers within a shift • Between 8 and 15 cases per interviewer within a shift, on average • Response rates lowest during the week, and especially before 5pm (41.1%) 10

  11. Analytic Approach • Examine interpenetration assumptions • Estimate ρint for each survey variable within each shift, based on respondent data • Test interviewer variance for significance • Estimate all variance components of the MSE of the respondent mean (possible with WDS data) • Estimate interviewer effects on (and interviewer contributions to) the variance components • Compute the proportion of interviewer-contributed variance due to NR error variance 11

  12. Example Derivation (Groves and Magilavy, 1984) • MSE of respondent mean (u = # of refusals): • Example: Refusal error component 12

  13. Example Derivation, cont’d • Estimate variance components using linearized variance estimators • Accounts for clustering due to interviewers and unequal workloads • Estimate interviewer effects on variances based on estimates of intra-interviewer correlations in true values (Census Bureau, 1985) 13

  14. Example Derivation, cont’d • Estimated total contribution of interviewers to refusal error variance: • The estimated contribution is a function of intra-interviewer correlations in true values, for respondents and refusals • Similar derivations for other components 14

  15. Results: Interviewer Variance Based on Respondent Data • Interpenetration evident in each shift • Four variable / shift pairs were found to have unusually large estimates of ρint: • Age at Divorce, Shift 2 (ρint = 0.08, p = 0.05) • Age at Divorce, Shift 3 (ρint = 0.10, p = 0.11) • Age at Marriage, Shift 2 (ρint = 0.11, p = 0.01) • Mths. since Marr., Shift 2 (ρint = 0.05, p = 0.13) 15

  16. Results: Sources of Interviewer Variance in Age at Divorce (Shift 2) • Estimated intra-interviewer correlations in • Response errors: -0.003 • True values for respondents: 0.092 • True values for refusals: 0.008 • True values for noncontacts: -0.055 • Total estimated variance of R mean: 0.423 • Total estimated variance contributed by interviewers: 0.033 • Additional variance contributed by interviewers is due to intra-interviewer correlations in true values for respondents! 16

  17. Additional Results • Similar findings for age at divorce in shift 3 • Response error variance was main contributor for age at marriage in shift 2 and months since marriage in shift 2 • Response error variance may arise from outliers, as shown in the following graph

  18. Illustration of Variance Sources

  19. Conclusions • Interviewer variance on key survey variables may arise from nonresponse error variance among interviewers • Interviewers may successfully obtain cooperation from different pools of respondents (e.g., older vs. younger) • Liking theory could be one explanation: variance in interviewer ages, voices  variance in respondent ages (F. Conrad) 19

  20. Implications for Practice / Future Work • Monitoring Strategies: managers can continuously compare available features of R and NR for each interviewer, and intervene when large differences arise • Findings need to be replicated in a face-to-face setting with interpenetrated subsamples assigned to interviewers • Access to interviewer features would also enable use of multilevel modeling 20

  21. Thank You! • Bob Groves, Mick Couper, Frauke Kreuter and Paul Biemer have provided very helpful feedback and comments • Thanks to Vaughn Call for providing access to the WDS data • Please email bwest@umich.edu for these slides or a draft of the paper 21

More Related