1 / 39

From face-to-face to web in a cross-national study

From face-to-face to web in a cross-national study. Ineke Stoop SCP/CST. ESS. Academic survey in 30+ European countries Attitudes, values and beliefs Biennial, start in 2002 1+ hour, face-to-face, effective sample size ~ 1500 Funders National Science organisations

stasia
Download Presentation

From face-to-face to web in a cross-national study

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. From face-to-face to web in a cross-nationalstudy Ineke Stoop SCP/CST

  2. ESS • Academic survey in 30+ European countries • Attitudes, values and beliefs • Biennial, start in 2002 • 1+ hour, face-to-face, effective sample size ~ 1500 • Funders • National Science organisations • European Commission, Framework Programmes • European Science Foundation • Management model • Central coordination • National implementation

  3. Aims ESS To monitor and interpret public attitudes and values within Europe and to investigate how they interact with Europe’s changing institutions Provide data on beliefs, attitudes and values for scientific and policy making purposes Measure attitude change in a changing Europe To advance and consolidate improved methods of cross-national survey measurement in Europe and beyond Collect data according to highest standards Generate methodological research Develop and disseminate new best practices Develop and improve social survey research infrastructure in Europe

  4. Content questionnaires ESS Everyround Roundspecific R1 Immigration Citizenship, involvement and democracy R2 Family, work and well-being Opinions on health and care- seeking Economic Morality R3 Personal & Social Well-being The Timing of Life R4 Ageism Welfare attitudes R5 Family, work and well-being in times of economic recession Trust in police and courts R6 Personal & Social Well-being Understanding, evaluation democracy • Socio-demo-graphic items • household composition • education • housing • occupation • social structure • Attitudes and behaviour patterns • religious affiliation and identity • ethnic and national identity • political and social trust • party affiliation and voting behaviour • media consumption • value orientations • social exclusion

  5. Access ESS Round 1-5 available Fully documented Free access Tables and graphics online, or download data Training package for young researchers Everything available All deviations documented

  6. ESS top data user countries (>50) (June 2012) (N=48,973)

  7. Methods: high quality • Strictly random sampling • No substitution, no proxy • Population: all inhabitantsaged 15+ • High target response rates • 70% target response rate, max. 3% noncontact • Face-to-face fieldwork • Gold standard, high response rates, universally applicable • Increasingly expensive • Rare in some countries • Pilots, pretesting and fieldwork monitoring • Analysis of cross-national differences • Nonresponse bias, measurement error, sampling design effects

  8. Methods: central coordination Central specifications for fieldwork Close monitoring national fieldwork (in advance and during fieldwork) Sampling design Translation procedures Guidelines response enhancement Guidelines interviewer training Contact forms, response calculation Context, events

  9. Can we comparecountries? Attitudes, values, beliefs Survey results Survey design, methodology, implementation

  10. Harmonisation strategies (Körner and Meyer, 2005) Harmonised concept Harmonised concept Harmonised concept Measurement procedure Measurement procedure Measurement procedure National survey/ Specific concept National survey/ Specific concept National survey/ Specific concept Input harmonisation Ex-ante output harmonisation Ex-post output harmonisation

  11. Output harmonisation • Ex post output harmonisation • Use existing sources • Ex ante output harmonisation • Set up new survey (or develop new instrument) • Output harmonisation • Best national quality • Or, national survey tradition • But, what about optimal comparability

  12. Output harmonisation: problems • Interview mode • Context question • Lay-out questionnaire • Question text • Introductory text • Answer scales • Handling don’t know • Size of boxes • Use of illustrations

  13. Input harmonisation • Input harmonisation • Design new survey • Optimal comparability • Or, are identical methods and instruments really equivalent in different countries? • But, what about optimal quality

  14. From face-to-face to web in the ESS • Mixed mode research and experiments • Different survey modes over time • Different survey modes across countries • Mixed mode within countries: different composition over time • Which modes: • Face-to-face • Telephone • Web

  15. ESS studies • Martin, Peter (2011) A Good Mix? Mixed Mode Data Collection and Cross-national Surveys. ASK Research & Methods, Vol. 20 (1, 2011) 5–26. • Martin, Peter (2011) What makes a good mix? Chances and Challenges of Mixed Mode Data Collection in the European Social Survey. CCSS Working Paper No. 02. • Martin, Peter, and Lynn, Peter (2011) The Effects of Mixed Mode Survey Designs on Simple and Complex Analyses. CCSS Working Paper No. 02. • Roberts, Caroline (2007) Mixing Modes of Data Collection in Surveys: A Methodological Review. ESRC National Centre for Research Methods. NCRM Methods Review Papers, NCRM/008.

  16. From face-to-face to web in the ESS • Coverage • Sampling • Nonresponse • Questionnaire design • Role interviewers • Measurement • Analysis • Cross-national comparison

  17. Coverage

  18. Internet access and broadband connections in households, EU27 (%)

  19. Individuals who used the internet at least once a week, 2011 (%)

  20. Individuals who used the internet on average at least once a week, by sex, age and education, EU27, 2011 (%)

  21. Sampling principles • Identical population definition • Complete coverage • No upper age limits • (Institutional population) • (Non-native language speakers) • Probability sampling • Effective sample size • Optimal sampling design in each country • Address, household, individual • Address and household: selection of target respondent in the field

  22. ESS R2 sampling frames (Häder and Lynn) Simple random sampling, or simple stratified sampling (no design effect)

  23. Comparison sampling frames • Achim Koch (GESIS) • In household and address samples: selection of respondent by interviewer • Leads to overrepresentation of women in household/address samples • Related to higher response rates • Close control on who answers the questionnaire in a household even more problematic in web survey

  24. ESS 1 – 4:Sample ofindividuals: Bias X response rate

  25. ESS 1 – 4:Sample ofhhs/addresses: Bias X response rate r = .44

  26. Nonresponse ESS • Response targets • 70% response rate • 3% noncontact rate • Standards in • Response rate calculation • Contact form • Nonresponse analysis

  27. Response efforts and monitoring • Guidelines response enhancement • Incentives/Brochure • Refusal conversion • Call schedule requirements • Minimum number of calls • Timing of calls (evening, weekend) • Contact form • Interviewer observations • Call records

  28. Response rates ESS (selection all rounds)

  29. Response rates over time

  30. Nonresponse • Response rate highest for face-to-face • Functional illiteracy (differs across countries) • Nonresponse analysis • What happens in web • Noncontact, refusal, not able • Reasons for refusal • Interviewer judgment • Refusal conversion

  31. Questionnaire design • Questionnaire length • Split questionnaire • Matrix design • Over time • Question design • Unimode • Developed for face-to-face

  32. Role interviewers • Select respondents (+) • Obtain cooperation (+) • Conduct interview • Explain questions (-) • Explain process (+) • Sensitive questions (+/-) • Validity answers (+) • Social desirability (-)

  33. Respondent understands questions

  34. Measurement • Mode effects • Part of analysis • No simple adjustment • Mixed modes: • Selection effects and mode effects • Always comparison face-to-face • Mode in analyses

  35. Cross-national comparison • Comparison over time • Mode change or time change • Comparison across countries • Mode difference or cultural difference • Mixed modes within countries • Selection effect or mode effect • Mode distribution over time • Feasibility of web for ESS (length, type of questions)

  36. Thank you for your attention

More Related