Surveys

1. Surveys Julie Demuth (With input from Jeff Lazo, Mary Hayden, other resources … and hopefully you!) Summer WAS*IS, July 14, 2007

2. Outline • What is survey research? • To use surveys or not: Some factors and information to consider • Survey research components  emphasis on design Remember, this is just a brief introduction!

3. What is survey research? • Surveys • Self-administered questionnaires (written) • Face-to-face interview, telephone interview, etc. (oral) • Cross-sectional design • More than 1 case, single point in time, analyzed to detect patterns of association

4. To use surveys or not? • Objectives • What information do I need? • What will I do with the information? • How will I analyze the information? • How and to whom will I present the results?

5. Advantages to surveys • Gathering information • Efficient for collecting large amount of information about a large population • Flexible to collect wide range of information (e.g., attitudes, values, beliefs, past behaviors) • Questions • Subjects gets same questions • Researcher can ask more complex questions • No response effect (willing to divulge more info w/o face-to-face contact; less likely to try to impress interviewer) • Administering • Relatively easy and inexpensive to administer • Can be computer-based

6. Disadvantages to surveys • Gathering information • Questions must be general enough to be appropriate for all respondents, possibly missing important context • Useless with non-literate or illiterate populations, only English-speaking populations, or hard-to-reach populations • Questions • Subjects’ motivation, memory, and ability to respond • Not appropriate for studying complex social phenomena • Subjects may not answer honestly • No control over participant interpretation • Administering • Low response rates • Respondents usually self-selected • Uncertainty about who actually filled out the questionnaire

7. Survey research components Design Sampling Implementation Analysis

8. Survey design • Parts of a survey • Introduction • Filtering questions • Content questions • Socio-demographic questions • Debriefing questions • Other considerations • Instructions • Formatting

9. The questions • Using focus groups • A la Mary Hayden! • Borrowing questions • Longitudinal studies • Developing them from scratch!

10. Criteria for assessing questions • Does the question require an answer? • If you read the area forecast discussion, does it help you better understand the weather? • To what extent does the respondent have an accurate, ready-made answer to the question? • What is your favorite episode of Friends? • Can people accurately recall and report past behaviors? • What time did you leave work on Dec 20, 2006?

11. More criteria • Is the respondent willing to reveal the requested information? • How many times have you driven through flooded roadways? • Will the respondent feel motivated to answer each question? • Scenario with 10 subparts

12. Possible scalar categories • Strongly agree  strongly disagree • Very favorable  very unfavorable • Extremely satisfied  extremely dissatisfied • None  Extreme • Scale of 1 to 5, where 1 means lowest possible quality and 5 means highest possible quality • Scales of 1 to 3, 1 to 7, 1 to 10

13. Question structures • Open-ended • Close-ended with ordered response categories (like scalar categories) • Close-ended with unordered categories • Partially close-ended • With “Other (please specify)_________” option

14. Principles for writing questions • Choose simple over specialized words • Avoid vague quantifiers when more precise estimates can be obtained • Avoid specificity that exceeds the respondent’s potential for having an accurate, ready-made answer • Use equal numbers of positive and negative categories for scalar questions

15. Principles for writing questions • Distinguish undecided from neutral by placement at the end of the scale • Avoid bias from unequal comparisons • State both sides of attitude scales • Avoid check-all-that-apply question formats

16. Principles for writing questions • Develop response categories that are mutually exclusive • Avoid asking respondents to say yes in order to mean no • Avoid double-barreled questions • Avoid asking respondents to make unnecessary calculations

17. Pretesting • Review by knowledgeable colleagues • Evaluate cognitive and motivational qualities • Verbal protocol analysis / think alouds • Retrospective interviews • Small pilot study • 10% of final sample size • Final check

18. How often do you get weather forecasts from the sources listed below?

19. More examples

20. More examples • I have a plan for keeping myself and my family safe in a tornado. • Yes • No

21. Survey sampling • Population, Units, Subjects and Samples • Population: entire group of people about which information wanted. • Units/subjects: Individual members of the population are called units • Sample: part of population examined • Error • Sampling: not surveying all elements of population • Coverage: not allowing all members of the survey population to have an equal or nonzero chance of being sampled

22. Survey implementation • Methods for survey implementation • Telephone • In-person • Internet • Knowledge Networks type of access • Mixed mode (e.g., telephone/mail) • Mail • pre-contact • cover letter • survey instrument • return envelope • incentive • reminder post-card • follow-up

23. Survey analysis (examples) • Univariate analysis • Histograms, tables, charts, etc. • Mean, median, mode • Range, standard deviation • Bivariate analysis • Crosstab/contingency tables • Correlations (Pearson’s r, Spearman’s rho, phi, Cramers V) • Regression analysis • Logistic, ordinal, linear, etc. • So much more!

24. Final thoughts • Many rules • Many resources • Beg, borrow, and steal NO such thing as a perfect study, survey, or sample!

25. So much more • Constraints -- Paperwork Reduction Act, Institutional Review Boards • Reliability and validity • Errors -- sampling, coverage, measurement, nonresponse • Biases

26. Resources • Dillman, D.A. 1999. Mail and Internet Surveys: The Tailored Design Method, 2nd Ed. John Wiley Company, New York: NY • Krueger, R. A. and M.A. Casey. 2000. Focus Groups: A Practical Guide for Applied Research. SAGE Publications • Presser, S., J. Rothgeb, M. Couper, J. Lessler, E. Martin, J. Martin, and E. Singer. 2004. Methods for Testing and Evaluating Survey Questionnaires. New York: Wiley and Sons • Tourangeau, R., Rips, L.J., and Rasinski, K. 2000. The Psychology of Survey Responses. Cambridge University Press. • http://www2.chass.ncsu.edu/garson/pa765/survey.htm • http://srcweb.berkeley.edu/index.html • http://www.jpsm.umd.edu/jpsm/index.htm • EACH OTHER!!