Download

Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with other approaches in ESS






Advertisement
/ 25 []
Download Presentation
Comments
blake
From:
|  
(1386) |   (0) |   (0)
Views: 88 | Added:
Rate Presentation: 1 0
Description:
Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with other approaches in ESS. Jaak Billiet: CeSO - K.U. Leuven Hideko Matsuo: CeSO – K.U. Leuven The European Social Survey Round 4 launching conference ‘Poland and Europe: continuation and change’.
Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with other approaches in ESS

An Image/Link below is provided (as is) to

Download Policy: Content on the Website is provided to you AS IS for your information and personal use only and may not be sold or licensed nor shared on other sites. SlideServe reserves the right to change this policy at anytime. While downloading, If for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.











- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -




Slide 1

Adjusting samples for nonresponse bias: pros and cons of surveying nonrespondents compared with other approaches in ESS

Jaak Billiet: CeSO - K.U. Leuven

Hideko Matsuo: CeSO – K.U. Leuven

The European Social Survey Round 4 launching conference ‘Poland and Europe: continuation and change’.

Institute of Philosophy and Sociology Polish Academy of Sciences, Warsaw 13 Jan 2010

Slide 2

Outline

  • Very short introduction

  • Short overview: Approaches to the assessment of bias applied in ESS (Billiet, Matsuo, Beullens & Vehovar, Research & Methods. ASK, vol 18 (1, 2009), pp. 3-43).

  • The surveys among nonrespondents (post R3): what, how, when, with what…?

  • Main results of NRS in PL and NO (comparison of results and focus on adjusting samples)

  • Pros and cons of NRS compared with other 3 approaches

Slide 3

1. Introduction

Analysis of nr bias still needed:

WHY? Still large differences in NR rates based on CF R4

Slide 4

2. Short overview(1)

  • Short overview of approaches to the assessment of bias applied in ESS (Billiet, Matsuo, Beullens & Vehovar, Research & Methods. ASK. vol 18 (1, 2009), pp. 3-43).

    In all rounds (R1, R2, R3, R4…..)

    • Bias as deviation between obtained sample and population(or ‘Golden standard’ survey) = post-stratification and evaluations of samples before and after weighting

    • Bias as difference between cooperative and converted refusals collected via refusal conversion = comparison of cooperative with reluctant respondents (converted refusals)

    • Bias as difference in ‘observable’ data among all sampling units (collected in contact forms)= sample based comparison between all respondents and all nonrespondents

Slide 5

Short overview (2)

In context of R3

4. Bias as difference between respondents and non-respondents collected via post hoc nonresponse survey= surveys among nonrespondents after R3 in PL, NO and CH (real NRS)

in BE (at moment of refusal only among refusals = Doorstep Questions Survey)

Slide 6

3. Survey among nonrespondents (1)

  • New survey among refusals with very small & easy questionnaire (some crucial variables) (Voogt, 2004; Saris)

  • Implemented in ESS Round 3 : 4 participating countries- Full mail survey (15 questions) months after main survey in NO (medium rr), CH (low rr) & PL (high rr)

    - At moment of refusal 7 crucial questions in BE (7 questions)

  • Response rates

    BE (44.7% = 303) response among refusals

    NO (30.3% = 342) response among noncontacts & refusals

    PL (23.2% = 192)

    CH (52.9% = 771)

    (cooperative much higher response)

Slide 7

Survey among nonrespondents (2)

1. The questions asked

Key questions procedure (Pedaksi approach)

Short 7 question module (+ at door): work situation, highest level of education, # of members in household, frequency of social activities, feeling (un)safe, interest in politics, attitude towards surveys

Normal 16 questions module: same as short + gender, year of birth, TV watching, voluntary work, trust in people, satisfied with democracy, trust in politics, immigration good/worse for country, (+ reasons for refusal (closed) in one subgroup)

Slide 8

Survey among nonrespondents (3)

2. Overview of the sample design

Slide 9

Survey among nonrespondents (4)

  • 3. Kinds of respondents in NRS

  • decisions to take in view of computing propensity scores for weighting the sample

  • NRS/(cooperative vs. nrs)

  • NRS/(cooperative vs. main)

  • (NRS+reluctant) vs (cooperative (nrs or main?))

  • NRS/cooperative vs reluctant/cooperative

  • see Figure next slide

Slide 10

Survey among nonrespondents (5)

Kinds of respondents in data analyses [NO, CH & PL]

ESS Reluctant

Respondent

(ESS3_Rrel)

ESS

ESS Cooperative

Respondent

(ESS3_Rco)

ESS Non-

Respondent

(ESS3_NR)

NRS Non-

Respondent

(NRS3_NR)

NRS Cooperative

Respondent

(NRS3_Rco)

NRS Reluctant

Respondent

(NRS3_Rrel)

NRS

Slide 11

Survey among nonrespondents (6)

Method used for adjusting the sample for nonresponse bias

1. Identify survey response differences on key explanatory variables between types of respondent (‘nonrespondent vs. cooperative respondent’).

2. Study neteffects of key explanatory variables on response probabilities via logistic regression model (dependent variable: prob ratio’s ‘nonrespondent/cooperative’).

3. Obtain propensity scores on all cases on non-response probabilities via logistic regression model (dependent variable: prob ratio’s ‘cooperative/nonrespondent’).

Slide 12

Survey among nonrespondents (7)

  • Transform propensity scores into weights via stratification method (Rosenbaum & Rubin 1984; Little 1986; Lee & Vaillant 2008):

    • Form 10 strata with equal number of cases after sorting on ps;

    • Assign each sample unit into correct corresponding sub-strata

    • Weight = expected probability/observed probability of the coop. respondent (or nonrespondent) in the corresponding sub-strata. 5.

Slide 13

Survey among nonrespondents (7)

5. Evaluate effects of propensity weighting via two main criteria:

  • Tests between unweighted & weighted sample on cooperative respondents (NRS3_Rco & ESS3_Rco). 1b. In case of significant differences: test differences between parameters of relevant substantive explanatory models

  • Study differences in distributions on key questions between types of respondents(NRS3_Rco vs NRS3_NR or ESS3_Rco vs. NRS3_NR).

Slide 14

4. Main results in NO and PL(1)

  • Differences between ESS cooperative and NRS !nonrespondents*

    * Only single ESS cooperative respondents (not ‘double’ respondents). All tests: ESS resp = expected freq

!

Slide 15

…differences in distributions

Main results in NO and PL (2)

!

Slide 16

2. Logistic regressionparameters nonresp/cooperative

Main results in NO and PL (3)

*NRS res are final ESS nonrespondents

Slide 17

(continued)Logistic regression parameters nonresp/cooperative

Main results in NO and PL (4)

Slide 18

Main results in NO and PL (5)

?

Main net effects on probability ratio coop resp / nonresp (inversed parameters!)

In Norway:probability of response INCREASES if

  • Higher educated

  • Participate more in social activities then most (subjective…)

  • More satisfied with democracy

  • Positive attitude towards ‘consequences’ of immigration

    In Poland: probability of nonresponse INCREASES if

  • Higher educated!!!

  • Unemployed

  • Feel safe

  • Participate less in social activities than most!

  • (political interested?!!)

Slide 19

Main results in NO and PL (6)

3. Evaluation of the propensity weights

First approach A: is the adjusted sample (weighted) of cooperative ESS respondents significant different from the original sample?

if yes: we may conclude that the adjustment had effect on the sample estimates

conclusion: no significant differences at all

example: variable with largest differences = education

Slide 20

Main results in NO and PL (7)

Differences between original sample and adjusted sample even smaller in PL

Not necessary to test a substantive regression model since the univariate distributions do not differ(first approach B)

This is nonetheless checked for model with “consequences of immigration” as relevant dependent variable” and number of predictor variables:age, TV watching, involvement in charity org, trust in politics, social trust, and two value orientations (conservation, self-transcendence)

R² = 0,26 in both models (not weighted & weighted)

all predictors contribute significantly to variance of dept. var

BUT: no differences at all between the two models

Conclusion = was ps weighting useless? Let us see the second approach

Slide 21

Main results in NO and PL (8)

Second approach:do the initial significant differences of belonging to a response category of all key questions between ESS respondents and nonrespondents (NRS res) in first table disappear after adjusting the sample of ESS cooperative respondents?

in other words, did we move from NMAR to MAR

let us see:

Slide 22

Main results in NO and PL (9)

Norway sample (Chisq values or t-values; p-values)

largely successful: all differences disappeared except political interest

Slide 23

Main results in NO and PL (10)

Poland: sample (Chisq values and p-values)

Not completely successful since still sign differencesbetween NRS and ESS for two variables (political interest and social participation)

Slide 24

5. Pros and cons of NRScompared with 3 other approaches

Slide 25

Conclusions

Future:

  • Other methods (contacting sequences using contact forms data) = expect low effect (result of some studies, see Blom)

  • More model based method: crucial is what additional information can be used

  • Combining different methods

  • Info in all methods = view on sensitive variables

  • Finally: low effect may mean LOW BIAS


Copyright © 2014 SlideServe. All rights reserved | Powered By DigitalOfficePro