Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with other approaches in ESS. Jaak Billiet: CeSO - K.U. Leuven Hideko Matsuo: CeSO – K.U. Leuven The European Social Survey Round 4 launching conference ‘Poland and Europe: continuation and change’.
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with other approaches in ESS
Jaak Billiet: CeSO - K.U. Leuven
Hideko Matsuo: CeSO – K.U. Leuven
The European Social Survey Round 4 launching conference ‘Poland and Europe: continuation and change’.
Institute of Philosophy and Sociology Polish Academy of Sciences, Warsaw 13 Jan 2010
Analysis of nr bias still needed:
WHY? Still large differences in NR rates based on CF R4
2. Short overview(1)
In all rounds (R1, R2, R3, R4…..)
Short overview (2)
In context of R3
4.Bias as difference between respondents and non-respondents collected via post hoc nonresponse survey= surveys among nonrespondents after R3 in PL, NO and CH (real NRS)
in BE (at moment of refusal only among refusals = Doorstep Questions Survey)
- At moment of refusal 7 crucial questions in BE (7 questions)
BE (44.7% = 303) response among refusals
NO (30.3% = 342) response among noncontacts & refusals
PL (23.2% = 192)
CH (52.9% = 771)
(cooperative much higher response)
1. The questions asked
Key questions procedure (Pedaksi approach)
Short 7 question module (+ at door): work situation, highest level of education, # of members in household, frequency of social activities, feeling (un)safe, interest in politics, attitude towards surveys
Normal 16 questions module: same as short + gender, year of birth, TV watching, voluntary work, trust in people, satisfied with democracy, trust in politics, immigration good/worse for country, (+ reasons for refusal (closed) in one subgroup)
Survey among nonrespondents (3)
2. Overview of the sample design
Survey among nonrespondents (4)
Survey among nonrespondents (5)
Survey among nonrespondents (6)
Method used for adjusting the sample for nonresponse bias
1.Identify survey response differences on key explanatory variables between types of respondent (‘nonrespondent vs. cooperative respondent’).
2. Study neteffects of key explanatory variables on response probabilities via logistic regression model (dependent variable: prob ratio’s ‘nonrespondent/cooperative’).
3. Obtain propensity scores on all cases on non-response probabilities via logistic regression model (dependent variable: prob ratio’s ‘cooperative/nonrespondent’).
Survey among nonrespondents (7)
Survey among nonrespondents (7)
5. Evaluate effects of propensity weighting via two main criteria:
* Only single ESS cooperative respondents (not ‘double’ respondents). All tests: ESS resp = expected freq
…differences in distributions
2. Logistic regressionparameters nonresp/cooperative
*NRS res are final ESS nonrespondents
(continued)Logistic regression parameters nonresp/cooperative
Main net effects on probability ratio coop resp / nonresp (inversed parameters!)
In Norway:probability of response INCREASES if
In Poland: probability of nonresponse INCREASES if
3. Evaluation of the propensity weights
First approach A: is the adjusted sample (weighted) of cooperative ESS respondents significant different from the original sample?
if yes: we may conclude that the adjustment had effect on the sample estimates
conclusion: no significant differences at all
example: variable with largest differences = education
Differences between original sample and adjusted sample even smaller in PL
Not necessary to test a substantive regression model since the univariate distributions do not differ(first approach B)
This is nonetheless checked for model with “consequences of immigration” as relevant dependent variable” and number of predictor variables:age, TV watching, involvement in charity org, trust in politics, social trust, and two value orientations (conservation, self-transcendence)
R² = 0,26 in both models (not weighted & weighted)
all predictors contribute significantly to variance of dept. var
BUT: no differences at all between the two models
Conclusion = was ps weighting useless? Let us see the second approach
Second approach:do the initial significant differences of belonging to a response category of all key questions between ESS respondents and nonrespondents (NRS res) in first table disappear after adjusting the sample of ESS cooperative respondents?
in other words, did we move from NMAR to MAR
let us see:
Norway sample (Chisq values or t-values; p-values)
largely successful: all differences disappeared except political interest
Poland: sample (Chisq values and p-values)
Not completely successful since still sign differencesbetween NRS and ESS for two variables (political interest and social participation)