The Survey Method Paul Lambert Applied Social Science, Stirling University 5.5.04, 9-11am

1. Nursing, Midwifery and Allied Health Professions Research Training Scheme Training Workshop May 2004 The Survey Method Paul Lambert Applied Social Science, Stirling University 5.5.04, 9-11am

2. Resources for this talk • Slides • the nature of survey research • issues in doing survey research • Reading guide • Activities sheet • introduction to research resources • example analysis of a secondary survey dataset NMAHP Research Training: Survey Method, May 2004

3. The Survey Method NMAHP Research Training: Survey Method, May 2004

4. 1. The Nature of Social Surveys NMAHP Research Training: Survey Method, May 2004

5. 1a) Surveys:The systematic collection of selected information from all or part of a population(see Marsh 1982) NMAHP Research Training: Survey Method, May 2004

6. Surveys are characterised by ‘variable-by-case matrix’ NMAHP Research Training: Survey Method, May 2004

7. Cases can be: • Any distinctive entity • Most often, they are individuals (people) • Variables are: • Measures of selected concepts of interest • Indicators (our ‘best guess’ at representing the concept) • Variable design: issues in choosing and formulating appropriate variables NMAHP Research Training: Survey Method, May 2004

8. Example: NMAHP Research Training: Survey Method, May 2004

9. The survey size • Total number of cases survey size (n) • A census covers every case in population. • Most surveys use samples of cases. • Larger survey size  more reliable sample estimates. NMAHP Research Training: Survey Method, May 2004

10. Varieties of social surveys • Topic: choice of variables / cases • Scale: number of variables / cases • Method: data collection format • Use: type of data analysis; descriptive v’s inferential NMAHP Research Training: Survey Method, May 2004

11. Why study survey research? • To undertake surveys • If answers research question; if attainable • Valuable skills • To understand / critique other people’s survey research based reports • Crucial – survey evidence is everywhere • Don’t just ignore / dismiss survey evidence NMAHP Research Training: Survey Method, May 2004

12. Strengths of surveys (1) • Can be representative / large scale • Probability theories justify generalisation from samples • Surveys can handle census or other large sample data collections and analysis • Parsimonious summary of the relation between variables on many cases NMAHP Research Training: Survey Method, May 2004

13. Strengths of surveys (2) • Extensive methods research • Eg, tests of reliability and validity - Bryman 2001 pp70-74: NMAHP Research Training: Survey Method, May 2004

14. Strengths of surveys (3) • Variety of data analysis formats • ‘Descriptive’, ‘Inferential’, ‘multivariate’ • Causal analysis defensible, eg longitudinal • Data analysis is falsifiable • Report writing skills, & careful qualifications NMAHP Research Training: Survey Method, May 2004

15. Strengths of surveys (4) • Accessibility of survey research • Most research questions benefit from survey investigation • Secondary datasets widely & freely available • Small scale surveys quick to conduct • Survey results oftenconvince others NMAHP Research Training: Survey Method, May 2004

16. Strengths of surveys (5) • Surveys are less biased than most other social research methods • Transparency of: sampling methods; variable construction; data analysis • Falsifiability • Cynicism of receiving audiences..! NMAHP Research Training: Survey Method, May 2004

17. Examples: surveys in health research • General purpose and focussed cross-sectional • Scottish Health Survey: self-reported health and lifestyle of 4000 adults • Selected population, eg sufferers of condition X • Longitudinal follow-up studies • Birth cohort studies: parental backgrounds and childhood health progressions • Ageing, status and sense of control (US): 1995 sample of 3k in 1995, recontact 1.5k in 1998 • Experimental designs • Smokers’ reactions to treatment programmes NMAHP Research Training: Survey Method, May 2004

18. 1. The Nature of Social Surveys NMAHP Research Training: Survey Method, May 2004

19. Role of sampling Surveys usually select only a sample of cases - aim to be representative of wider population • Key idea is inference = confidence in our ability to generalise Sampling inference = application of statistical theories in order to estimate probabilities that a sample result is ‘likely to have been unrepresentative’ NMAHP Research Training: Survey Method, May 2004

20. The ‘normal’ (Gaussian) curve NMAHP Research Training: Survey Method, May 2004

21. Theories of sampling methods Sampling and probability theories tell us that any particular random sample is most likely to have the same properties as the wider population. We can then estimate the probability that sample results of a particular nature could have arisen by chance, rather than because they are the same as the (unknown) population result. NMAHP Research Training: Survey Method, May 2004

22.  If the cases in sample surveys were selected at random, then can use sampling theories and thus ‘inference’ NMAHP Research Training: Survey Method, May 2004

23. Statistical inference ..causes confusion; one of hardest parts of survey data analysis to understand.. Phrases: ‘significance level’ ‘p-value’, ‘confidence interval’, ‘hypothesis testing’, .. Meaning:Whether results would probably generalise to a larger population (if sample is treated as random) See: Refs on reading list (esp Wright 2002) NMAHP Research Training: Survey Method, May 2004

24. ‘Inferential data analysis’ • Variable-by-case matrix data analysis for generalising findings to population • Often distinguished from ‘descriptive’ data analysis (results of sample only) • Key: joint influence of • 1) size of sample • 2) strength of data pattern in increasing confidence about generalisations NMAHP Research Training: Survey Method, May 2004

25. Doing good inferential analysis is difficult: • Reliable sampling resources expensive • Many early critiques of survey research concerned inappropriate inferential analysis • Contemporary survey research tends to follow 2 alternate strategies: Large scale, often secondary, rigorous inferential methods or Small scale, primary, claims carefully qualified NMAHP Research Training: Survey Method, May 2004

26. Drawing samples (case selection) NMAHP Research Training: Survey Method, May 2004

27. Sampling methods = Ways of selecting case from population NMAHP Research Training: Survey Method, May 2004

28. a) Simple Random Sample • A statistical method used to choose cases randomly (eg random numbers) Every case in population has exactly the same chance of being in sample • Most data analysis techniques initially designed for simple random samples NMAHP Research Training: Survey Method, May 2004

29. b) Systematic Random Sample • Like the Simple RS, select cases from anywhere in the whole population • An easier selection method : choose every (n)th person for the sample • Danger of ‘periodicity’ if original population order has any structure,  bias NMAHP Research Training: Survey Method, May 2004

30. Problems with sample methods selecting from whole population • The ‘random’ part means it’s always possible to get a population coverage quite different from known structures • If total population is large or dispersed, then coverage of random parts of it is expensive and time consuming: few surveys use random sampling from whole of UK NMAHP Research Training: Survey Method, May 2004

31. c) Stratified random samples • Modifies random sample to ensure intended coverage of population groups • split sampling frame by stratification factors • select random samples within each factor • final sample has fixed proportions of each • Example: select 490 M and 510 F • Properties: proportionate sample; correct representations; but more expensive & complex; may need ‘weights’ for analysis NMAHP Research Training: Survey Method, May 2004

32. d) Multistage cluster samples • i) Select clusters of population at random • ii) Sample randomly within clusters • Eg: clusters = local authorities in UK • With qualifications, may still be treated as ‘random’ for analysis purposes • Big reduction in costs if face-to-face contacts  Most widely favoured sampling method in large scale survey collections NMAHP Research Training: Survey Method, May 2004

33. Example: Multistage cluster sample • Interest: attitudes of Scottish school pupils • Resources: 400 interviews with pupils NMAHP Research Training: Survey Method, May 2004

34. Shetlands 2 Highlands 40 Islands 20 Moray 20 Aberdeen 40 Perth 20 Edinburgh 100 Argyll 24 Borders 10 Glasgow 124 NMAHP Research Training: Survey Method, May 2004

35. Moray 40 Stirling 60 Edinburgh 150 Glasgow 150 NMAHP Research Training: Survey Method, May 2004

36. Stirling 60 30 young people at Balfron School and 30 young people at Stirling High NMAHP Research Training: Survey Method, May 2004

37. Issues in random sampling • Only as good as underlying sampling frame(a good one may not be available, or not be as good as we think) • Data analysis methods need adapting for stratified / clustered designs • Other survey factorsinteract with sample selection issues, eg poor interviewers may discourage certain cases from response NMAHP Research Training: Survey Method, May 2004

38. ii) Opportunistic sampling • Often in social research, sample design is ‘opportunistic’ (‘purposive’) • Random sampling is expensive • Random sampling is complex • Some purported random samples are actually purposive anyway (understanding ‘random’) NMAHP Research Training: Survey Method, May 2004

39. a) Quota sampling • Fill up quota’s of groups of interest • Quota’s can ensure: • overall representation (cf systematic) • broad topic coverage (eg types of voter) • Example: market researchers in street; telephone call centres vetting contacts • Biasses: issues in how a quota ‘fills up’ NMAHP Research Training: Survey Method, May 2004

40. b) Snowball sampling • Also ‘focussed enumeration’ • Technique for contacting cases from populations rare / difficult to access • Ask first obtained contact for suggested further contacts  snowball gathers size… • Eg – smaller ethnic minority groups • Problem: social networks are non-random! NMAHP Research Training: Survey Method, May 2004

41. c) Convenience sampling • Samples whatever cases from population were easiest to reach, eg personal contacts • Often no other sampling strategy involved • Biasses likely in convenience process • Examples: …most student survey projects are ‘convenience’..! NMAHP Research Training: Survey Method, May 2004

42. Random v’s Opportunistic • Random sampling difficult and expensive – mainly government funded surveys • Much data analysis / inference assumes random sample, but not applied to • But random sampling is not a panacea... • And opportunistic data is often robust… Rule: Use survey documentation to report sampling process and any errors NMAHP Research Training: Survey Method, May 2004

43. 1. The Nature of Social Surveys NMAHP Research Training: Survey Method, May 2004

44. Varieties of social survey designs • Micro-social data: • Census’s • Cross-sectional surveys • Longitudinal surveys • Cross-nationally comparative surveys • Experiments or ‘quasi-experiments’ • {Macro-social data} • Single summary statistics describing outputs {from survey analyses} NMAHP Research Training: Survey Method, May 2004

45. Census’s • General overview of whole population • ‘Disclosure risk’ issues • Cross-sectional surveys • Very widely used format • Huge range of topic coverage • Often used to study particular or rare subpopulations NMAHP Research Training: Survey Method, May 2004

46. Longitudinal datasets :studies involving time • Repeated cross-sections • Chart changes over time, eg yearly means • ‘Panel’ and ‘cohort’ samples • recontact an initially random sample • Learn about changers and causes of actions • Problems of attrition • Retrospective sample • Rely on recall evidence of random selection • Problems of selective recall • Strengths: understand process and causality • Problems: sampling and attrition; complexity NMAHP Research Training: Survey Method, May 2004

47. Cross-nationally comparative datasets • Focussed surveys (IPUMS census’s; ISSP; World Values Survey; European Social Survey) • Longitudinal studies (LIS; ECHP; CHER) • Many analytical attractions, but issues of comparable analysis are complex NMAHP Research Training: Survey Method, May 2004

48. Experimental / quasi-experimental designs • Experiment: researcher intervenes in the process of study (quasi-experiment: ‘observe’ intervention) • Dream of the randomised controlled trial • Rare in sociology: cost & ethics; morecommon in psychology & certain health research fields • Consequences: • Different methods of analysis (see eg Robson 2002 c5) • Less concern over inference / v large samples NMAHP Research Training: Survey Method, May 2004

49. Simple and complex survey data • Simplest variable-by-case matrix has one sample of independent cases from 1 period • More interesting social science data has more complex designs, eg… • Multiple records per case • Relations between cases • Experimental matching of designs NMAHP Research Training: Survey Method, May 2004

50. Working with complex survey data.. • Advanced tasks in data management • File matching • Variable transformations and treatments • Advanced methods of data analysis • Complex findings not easily summarised or communicated • …but it is simplicity of simple survey data that many people criticise about SDA… NMAHP Research Training: Survey Method, May 2004