Effective Survey Design in Social Research

1. Structured Surveys in Social Research

2. What are surveys? • One method of collecting, organising analysing data • Data collection - is structured and systematic • Info. on same variables from multiple cases (people, groups, orgs.) • Both quantitative and qualitative data • Data analysis - looking for systematic variation between variables and cases • Variable by case matrix form – like experimental method • But more difficult to attribute differences to experimenter intervention • Multiple techniques Questionnaires most common; • But also: structured and in-depth interviews, direct observation techniques etc.

3. Survey v Participatory methods

4. The Steps in a Survey Project • Establish the goals of the project - What you want to learn • Determine your sample - Whom you will interview • Choose interviewing methodology - How you will interview • Create your questionnaire - What you will ask • Pre-test the questionnaire, if practical - Test the questions. • Conduct interviews and enter data - Ask the questions. • Analyze the data - Produce the reports.

5. Descriptive and Explanatory Research 1 • Survey research has two fundamental questions: • What? – Descriptive • i.e. Social structure, market research, opinion poll • Why? – Explanatory • Theorising • Grounded / ex post facto theory • Efficient data collection and analysis

6. Descriptive and Explanatory Research 2 • Descriptive research – focusing the topic • Time frame? • Geographical location? • Comparison of sub-groups? • Explanatory research • Either causes of change • And / or consequences of change? (i.e. PIM) • Units of Analysis • Village / Household/ Individual

7. Farming Systems and Units of Analysis

8. Theory Construction & Testing Obs1 Obs2 Obs3 Obs4 Empirical level Theory Construction Theory Conceptual abstract level Theory Testing Obs1 Obs2 Obs3 Obs4 Empirical level

9. Survey Design 1 • Classic design – experimental v control group • Availability of controls? • Repeated measures for same group? • Panel design – same group over time • Longitudinal survey • Problem of re-visiting all panel members? • Quasi-panel design – diff groups over time

10. Survey Design 2 • Retrospective panel or experimental design • Based on recall – selective memory? • Cross-sectional design – most common • Matched experimental and non-experimental groups • Comparison at one point in time • One shot ‘case’ study – most primitive • One group at one point of time • Limited use for evaluating causal processes

11. Sampling • Two main types of sample • 1. Non-probability samples • Some cases have greater or unknown selection chance • 2. Probability samples to achieve representative sample of population to permit generalisation • Each case has equal or known chance of selection • Random selection from sampling frame • Sampling error – standard error

12. Probability Samples 1 • Four types – choice depends on • Resources - Desired accuracy, survey method and availability of a good sampling frame. • 1. Simple random sampling (SRS) • Determine sample size • Random number table selection • Requires good sample frame • 2. Systematic sampling • Like SRS but simpler – same limitations • Proportional sampling i.e. every fifth member of frame – periodicity?

13. Probability Samples 2 • 3. Stratified sampling • Select stratifying variable(s) i.e. wealth • Proportion in different groups same as population • Requires a priori knowledge of stratifying variable • 4. Multi-stage cluster sampling • Area based i.e. random selection of districts, streets and finally households

14. Sample Size? • Size depends on two key factors: • Degree of accuracy required (max 5% SE?) • Population variability re. key study variables • For small samples, a small increase in sample size can result in a large increase in accuracy • Variability of most heterogeneous variables? • Requirements of statistical techniques? • Allow for non-response – be aware of bias!

15. Non Probability Samples • Where no reliable sampling frame is available • Or where populations are highly dispersed • Or where no requirement for generalisation from sample to population • Exploratory analysis • Hypotheses generation • Testing questionnaires • Purposive, quota and availability sampling

16. From Concepts to Questions • Descending the ladder of abstraction! • Clarify concepts – functional v substantive • Develop good indicators • Measurable: No’s & dimensions (space and time) • Reliability – Consistency in repeated use • Validity – Ability to measure the concept as intended • Consistency of meaning to different people ! • Pilot test or borrow from existing questionnaires • Clarify concepts and indicators with iteration

17. Explanatory variables • Dependent variables – i.e. farmers level of relevant knowledge of information and skills and experience. • Independent variables –i.e. farmers learning & info. gathering methods, socioeconomic status, ethnicity, etc. – factors which might explain their knowledge of the dependent variables • Confounding variables – Other factors related to both dependent and independent factors which may distort the results and have to be adjusted for.

18. Open or Closed Format Questions? Open formatquestions • Allows exploration of the range of possible themes arising from an issue • Can be used even if a comprehensive range of alternative choices cannot be compiled • Closed or forced choice-format questions • Easy and quick to fill in • Minimise discrimination against the less literate or the less articulate • Easy to code, record, and analyse results quantitatively • Easy to report results

19. Design and Response Rates • Short simple questionnaires usually attract higher response rates than long complex boring ones. • Adding variety in the types of questions asked will help • Make and give incentives? • Ordering of questions • Go from general to particular. • Go from easy to difficult. • Go from factual to abstract. • Start with closed format questions. • Start with questions relevant to the main subject. • Do not start with personal questions.

20. Other considerations • Avoid leading and double barrelled questions • Keep language simple (short) and precise • Frame of reference: who and over what time period • Anticipate interpretational difficulties • By respondents and enumerators – piloting • Appropriate questions – cultural sensitivity • Direct and indirect questions • Be aware of prestige bias – i.e. wealth ranking

21. Cultural Sensitivity?

22. Administering the questionnaire Advantages of self administered questionnaires • Cheap and easy to administer. • Preserve confidentiality • Can be completed at respondent's convenience • Can be administered in a standard manner Advantages of interview administered questionnaires • Allow participation by illiterate, less well educated people. • Higher response rate?? • Allow clarification of ambiguity through dialogue

23. Examplesfrom Sri Lanka

24. Surveys used to measure fish yields from stocked village reservoirs • Second Phase: 5 reservoirs belonging to 4 villages stocked, 1 control village • 1. Direct observation • Collective fishing and staggered harvesting • 2. Longitudinal panel survey • 7 day recall of household fish consumption • Fortnightly, 41 wealth stratified households • 3. Retrospective panel survey • Participatory impact monitoring (PIM) • Stakeholder perceptions and yield recall

25. 1. Longitudinal Survey -Mean monthly per capita consumption by source

26. Monthly Yield by Species & Tank

28. Data Management

29. Data Management • Spreadsheets: Lotus, Excel etc. • Regular ‘symmetrical’ data matrices • Arithmetic calculations • Databases: Access, FoxPro, File-Maker, Oracle • Relational Design - 3D v 2D :- • Large, irregular and longitudinal surveys • Data Accessibility and exploratory analysis • Non-numerical Databases i.e. Nudist, InVivo • Statistical packages i.e. Minitab, SPSS

30. Spreadsheet Approach

31. Database Relational Design Primary Key Primary Key ‘One to many’ relationships

32. Database Benefits 1 • Ease of Data entry • Organised ‘compact’ tables • Automated coding systems • Entry forms look like questionnaire formats • Multiple Users - online (web pages) or offline synchronisation • Data entry error checks • Through relational design • Other built in validation steps

33. Database Benefits 2 • Data Analysis • Demands a priori planning • Exploratory Analysis before higher level statistical analysis • Data reduction - cross-tabulation (Pivot Tables) • Integrated with MS office suite and statistical packages - SPSS, Minitab…

34. Limitations • Exponential learning curve • Dependence on technical backup support • Fallback strategies • Structured design Limits flexibility / iteration possibilities • Empiricism v anthropological view of human behaviour as complex sets of meanings