Methodological Rigour in Survey Research

1. Methodological Rigour in Survey Research A case study of the European Social Survey Michael Breen

2. Issues in Survey Research • Questionnaire design • Sampling strategy • Fieldwork oversight • Response rates • Cross national research issues

3. European Social Survey (ESS) • The European Social Surveywas established in 2001 as an academically-driven social survey designed to chart and explain the interaction between Europe's changing institutions and the attitudes, beliefs and behaviour patterns of its diverse populations. • Currently in the midst of its sixth round, this biennial cross-sectional survey covers more than thirty nations and employs the most rigorous methodologies. • All data, questionnaires, an interactive analysis tool (NESSTAR) and an educational resources for use in HEIs are all available online at http://www.europeansocialsurvey.org/.

4. Questionnaire design • The core questionnaire items cover both socio-demographic and substantive themes. The content of the core remains largely constant at each round. It includes both independent and dependent variables; the latter designed to measure shifts over time in what are considered to be key components of Europe’s social fabric. • These core questions have been designed in collaboration with a group of experts in different fields. This is overseen by the Scientific Advisory Board of the ESS. • Translations are required for each language used as a first language by 5% or more of the population.

5. Sampling strategy • Persons aged 15 years and over who are resident in private households in the Republic of Ireland • Must be a randomly derived sample • Must conform to the ESS sampling specification • Strategy must be agreed with the field agency and signed off by the Core Scientific Team of ESS before fieldwork begins.

6. Margin of Error (95% Confidence Interval)

7. Sample Design in the ESS (3 stage) • Stage 1: Selection of 225 PSUs (i.e. clusters of addresses) from a national total of 3,123 clusters formed from the GeoDirectory. Clusters are sorted geographically within eight strata (high/low elderly population; urban/rural; high/low education). Clusters are selected using systematic sampling with a random start. The probability of selection for the clusters is proportional to the number of addresses in the cluster. The minimum cluster size is 500 residential addresses. • Stage 2: Systematic sample of 20 addresses in each cluster. Within each cluster, addresses are sorted geographically. • Stage 3: Within dwellings, the next birthday method will be used for equal-probability selection of one person who is 15 years old or over, with no upper age limit within each household. No substitution of respondents will be permitted

8. Effective sample size • Most statistical tests assume data is obtained by simple random sampling. • If random sampling is not used, the traditional statistical tests become incorrect to an unknown degree. • The sample needs to be corrected to include design effects • There is a difference between the effective sample size and the actual sample size.

9. Design effects • Design effect due to clustering: • DEFFc= 1.31= 1+ (10.68-1)* 0.0321. • Design effect due to different selection probabilities within households (from population statistics, updated to Q4 2009): DEFFp= 1.212 • Total design effects • DEFF = 1.59= 1.31 * 1.21.

10. Sample Size for ESS Round 5 • Target response rate is 60% for Ireland from previous experience • The gross sample size must be 4500. With 11% of ineligibles and a response rate of 60% the net sample size must be 2,403. • N_net = (4500 * 0.89 * 0.6) = 2,400 • For each cluster 20 addresses have to be drawn. That results in an average of 10.68 interviews per cluster. With an overall DEFF of 1.59, that results in an effective sample size of 1515. • N_eff = (2405 * 1.59) = 1515

11. Fieldwork Agency & Oversight • Contract based on tender document • Tender handled at European level • 90+ Expressions of Interest • 4 tenders received • Agency liaison (via postdoctoral fellows) • Clearly defined deliverables • Weekly reporting • Quality control back checks • Coding issues/data format • Billing control

12. Deliverables: Data files • 1. Main and Supplementary questionnaire, with additional ESS variables • 2. Interviewer questionnaire • 3. Contact form • 4. Parents' occupation • 5. Raw data • 6. Sample design data file (SDDF)

13. Deliverables: Documents • National Technical Summary • Population statistics • Main questionnaire • Supplementary questionnaire • Interviewer questionnaire • Contact form • Show cards • Interviewer and fieldwork instructions • Advance letters, brochures and other written information to the respondents • CAPI programmes, if applicable

14. Response rates enhancement • Four personal visits by interviewers to each sampling unit on different days of the week and times of day, of which at least 1 must be at the weekend and 1 in the evening. These visits should be spread over at least two different weeks • In order to allow difficult-to-contact people to be located, the fieldwork period can not be less than 30 days • Use of incentives • Provision of telephone numbers for potential respondents to contact • Extra training of interviewers in response-maximisation techniques and doorstep interactions, • Implementing refusal avoidance and conversion techniques • Re-issuing of refusals and non-contacts • Refusal conversion, i.e. obtaining the cooperation of initially reluctant sample persons

15. Response rate calculation • The target for the ‘ESS’ response rate is 70%. • The Irish rate was set at 60% from previous experience • It is calculated as Response rate = number of achieved interviews number of individuals – ineligibles • Achieved response rates in ESS Ireland Round 5 was 65.4%, the highest rate ever achieved in the ESS in Ireland. • (Highest in Round 5 was Bulgaria at 81%, lowest Lithuania at 40%

16. Cross national research issues • For validity, multinational surveys have to be based pm probability sample designs of comparable national populations. • All measurements should be well controlled for comparability. • Translations must be accurate and comparable • Significance of “media events” must be noted • Multilevel modelling is required using hierarchical linear regression • Structural equation modeling is also used to build causal pathways on a country by country basis. • While the software is becoming more plentiful and more powerful (AMOS, MLWiN, R, Rasch) it requires clearly defined theoretical foundations before analysis.