Robert Voogt Dutch Ministery Of Social Affairs and Employment (formerly of the University Of Amsterdam)

1. Nonresponse in survey research: why is it a problem? Robert Voogt Dutch Ministery Of Social Affairs and Employment (formerly of the University Of Amsterdam)

2. Overview • What is nonresponse, why is it a problem and why does the traditional way of correcting for nonresponse not solve the problem • Overview of general correction techniques • An alternative approach to correct for nonresponse bias • Real life illustration

3. What is nonresponse, why is it a problem and why does the traditional way of correcting for nonresponse not solve the problem?

4. Survey research • Population is sampled • Sample is a good representation of population when good sample techniques are used • Not all sample elements will respond

5. Unit vs Item nonresponse • Some are not reached, others refuse or are not sending back the questionaire: unit nonresponse • Some who do answer the questionnaire do so incompletely: item nonresponse

6. MCAR, MAR, MNAR 3 general nonresponse mechanisms can be distinguished • MCAR: Missing Complety At Random • MAR: Missing At Random • MNAR: Missing Not At Random

7. Missing Completely At Random (MCAR) • Conditional distribution M given the survey outcomes Y and survey design variables Z. Let f(M|Y,q) denote the distribution, with q the unknown parameters. • If MCAR: f(M|Y,Z,q) = f(M|q) for all Y,Z,q • Not a realistic assumption

8. Example MCAR • Taking a random subsample of a group of nonrespondents • If random subsample of nonrespondents is analysed (after obtaining answers of all of them), the nonsampled nonrespondents can be said to be MCAR • So correction methods using the MCAR assumption can be used

9. Missing At Random (MAR) • MAR: f(M|Y,Z,q) = f(M|Yobs,Z,q) for all Ymis,q • where Yobs denotes all the observed survey data • This means that missingness depends on the observed variables, the observed values of incomplete variables or on the design variables, but not on the variables or values that are missing

10. Example MAR • For both respondents and nonrespondents we know their level of education • Respondents who share the same value of level of education have the same distribution on the unobserved variables • Most survey nonrespondent adjustment methods assume MAR

11. Not Missing At Random (NMAR) • NMAR: f(M|Y,Z,q) = f(M|Yobs,Ymis,Z,q) for all Yobs,Ymis,q • This means that missingness depends on missing values after conditioning on the observed data • To get an unbiased distribution M, a joint model of the data and the nonresponse mechanism is necessary

12. Example MNAR • For both respondents and nonrespondents we know their level of education • Given the level of education nonresponse on the variables of interest is not random • This means it is not sufficient to use only level of education to correct for nonresponse bias.

13. Nonresponse bias If nonresponse is not a result of design, almost always NMAR is the case, with data biased by nonresponse as a result. The amount of nonresponse bias is dependent on: • the correlation between the target variable(s) and the nonresponse mechanism; • the level of nonresponse.

14. Nonresponse bias with Yk: the score of element k in the population on the target variabele rk: probability of response of element k in the population when contacted in the sample C(r,Y): population covariance between response probabilities and the values of the target variable

15. Nonresponse bias with • (Yk-Y): the difference between the population score and the score of element k on the variabele of interest • (rk-r): the difference between the mean probability to respond and the probability to respond of element k • It follows from this equation that the response level in itself does not say everything: the amount of bias depends on the relation between the first and second part of the equation

16. Traditional correction methods • Use population information to compare to the respondent group with the population • Use information that is available for both respondents and nonrespondents • Use information about the difficulty to obtain data from the respondents • In fact, the assumption is that the data are MAR, given the values of the variables of which population information or information about the nonrespondents is available

17. Traditional correction methods • No information about the difference on the variables of interest between the respondents and nonrespondents • No information about the difference in response probabilities between sample elements that score different on the variables of interest • So there is no reason why this way of correcting should work

18. Overview of general correction techniques

19. Different correction techniques • Weighting: assigning each observed element an adjustment weight • Extrapolation: respondents who are most like the nonrespondents are used for correction • Imputation: missing values are substituted by estimates

20. Weighting • Weighting: assigning each observed element with an adjustment weight • Sample elements that belong to groups that seem underrepresented on the variables used in the weighting will have a high adjustment weight • Sample elements that belong to groups that seem overrepresented among the respondents will have a low adjustment weight

21. Weighting Example • Question: Have you ever visited Lugano? (Y/N) • Population information available about age (18-30 ) (31-64 ) (65-older) • Comparison of respondents and population • Weighting Yes: 4*1.5 + 35*.7 + 1*2.0=6+24.5+2=32.5 No: 16*1.5 + 35*.7 + 9*2.0=24+24.5+18=66.5

22. Extrapolation • Central idea: some groups of respondents are more like the nonrespondents than others are • For example, sample elements that first refused, but when contacted for the second time, were persuaded to participate, can be used as proxies for the final refusals

23. Extrapolation Example • Question: Have you ever visited Lugano? (Y/N) • Two respondent groups: early respondents and late respondents • Calculate the distribution among the nonrespondents using the last respondent method Last respondent: L=A2+(A2-A1) (X2-X1/X2), with: L: theoretical last respondent A: % response to an item in a wave X: cumulative % respondents at the end of a wave L = 50+(50-40) (67-40/67) = 50+*.40=18%

24. Imputation • Imputation: missing values are substituted by estimates Different methods of imputation: • Single Imputation: for each variable one value is imputed • Hot Deck Imputation: a missing value is replaced by an observed value of a comparable respondent • Multiple Imputation: for each variable several values are imputed; in this way the uncertainty that imputation brings with it is also taken into account

25. Hot Deck Imputation Example • Divide the respondents into homogenous groups. For exampe, by using CHAID. • CHAID recursively partitions a sample into groups so that the variance of the dependent variable is minimized within groups and maximized among groups • Link each nonrespondent to the group it fits in best • Substitute the values of a random respondent from the same group as the value of the nonrespondent

26. Hot Deck Imputation Example, part 2 CHAID finds groups: age 18-30, 31-64/low education, 31-64/high education, 65+/male and 65+/female

27. Multiple Imputation Example • For each case, 5 values for each missing variabele are calculated, using a regression equation and adding a random error term • These values are combined in one single value, for example, by taking the mean • The variance will take the uncertainty due to the imputed value into account by combining the within imputation variance (the variance of each estimated data set) and the between imputation variance (in which all 5 data sets are used)

28. Multiple Imputation Example, part 2 Percentage that has visited Lugano

29. An alternative approach to correct for nonresponse

30. Key to succes of correction methods • The information used in the correction method • The correction method must model the nonresponse mechanism • The variables used in correction should have a relation with: • the variables of interest • the probability to respond of a sample element

31. Central Question Method(Betlehem & Kersten, 1984) • Nonrespondents are asked to answer one (or more) questions central to the subject of the study • The central questions are believed to have a strong relation with both the nonresponse process and the subject of the study • Central questions are used in correction

32. Central Question Example • Central Question: Have you ever visited Switzerland? (Y/N) • Question of interest: Have you ever visited Lugano? (Y/N) • Comparison of respondents and non-respondents • Weighting as correction technique Yes: 40*.72 + 0*1.43 = 28.8 + 0 = 29 No: 20*.72 + 40*1.43 = 14.4 + 57.2 = 71

33. Real Life Illustration

34. Illustration • Election study • High levels of nonresponse • External information available to test the succes of the correction procedures

35. Our research questions • Does nonresponse causes a problem in election studies? • Is using background variables sufficient or do we need central questions? • Do different correction techniques lead to different results? • Is it really necessary to recontact nonrespondents?

36. Data Collection • City of Zaanstad, The Netherlands • N=995; 901 used • Recontacting refusals • Mixed mode data collection • Two central questions: • Voted in 1998 national elections • Political interest

37. Response rate

38. Does nonresponse cause problems? We distinguish four groups: • Response at first contact (470) • Response after two contacts (76) • Response after three or four contacts (158) • Nonrespondents (including those who answered the central questions) (197)

39. Comparison of response groups Conclusion: nonresponse bias is present

40. How to correct? Using the Central Question Procedure and compare it with more traditional correction methods Two central questions: • Voted at national elections (0-1) – from election lists (so no response bias) • Political interest (0-1) – from short nonresponse questionnaire

41. Correction methods • Weighting by background variables / + central questions • Extrapolation • Hot Deck Imputation by background variables / + central questions • Multiple Imputation by background variables / + central questions for response levels of 52 % and 78 %

42. Weighting • On background variables: age, ethnicity, gender, household composition, education, residential value, number of years living in current residence, social cohesion in neighborhood; using an iterative procedure • As above plus validated voter turnout national elections 1998 and political interest (central questions)

43. Extrapolation • Last Respondent Method

44. Hot Deck Imputation • Obtain subgroups by using CHAID • Assign nonrespondents to the groups • Decide exact value to be imputed using a regression model (multiple imputation) • For background variables / background variables and central questions

45. Multiple Imputation • Use AMELIA (King et al., 1998) to calculate 10 discrete imputation values for each variable • Calculate the mean distribution by summing the 10 proportions of each of the categories of the variable and divide it by 10 • Compute variance to take both within- and between-imputation variance into account • For background variables / background variables and central questions

46. Dependent variables • Voted at national elections • Voted at provincial elections • Self-reported political interest • Importance of voting

47. Results for weighting Rsp: Respondents, BG: Background variables CQ: Central Questions, TS: Total sample

48. Compare different methods Rsp: Respondents, W: Weighting, HDI: Hot Deck Imputation, MI: Multiple Imputation. EX: Extrapolation, BG: Background variables, CQ: Central Questions, TS: Total sample

49. Relations: regression turnout provincial elections

50. Conclusions • Using cental questions lead to better estimates than only using background variables • Higher response levels lead to better estimates • All correction techniques perform equally well: the information used in the correction is more important than the technique used • Correcting bias in regression parameters is less succesful