Surveys!

1. Surveys!

2. What are the differences between surveys, interviews, scales, and questionnaires?

3. Issues with writing items • Determine content and purpose of question • Choose the response format • Figure out how to word it • Figure out where to put it • Pilot test! • Ask for feedback from participants (at least have a comments box)

4. Question content • Is the question needed? At that level of detail? • Is there a double-barreled question? Do you need to ask more than 1 question? • Do p’s have the info needed to answer the question? • Do you need to be more specific or more general? • Are there biases in the question? • Will people answer the question honestly? • Hypothetical projective respondent

5. Make sure each item has only 1 issue I feel completely secure in facing unknown new situations because I know that my partner will never let me down.

6. Types of responses • When would you want to use dichotomous vs. nominal, ordinal, or interval data? Advantages and disadvantages? • In a Likert scale, how many options should you have? • How should they be set up? • Should there be a neutral point? • How can the responses you offer affect results?

7. More on responses • When would fill-in-the-blank be good or bad? • When should you give multiple options? • Do you have all the alternatives without going into too much detail? • When are unstructured response formats good?

8. Example from a survey How often do you exercise? Infrequently 17% Occasionally 48% Often 35% In the last six months, how often have you engaged in at least 20 minutes of aerobic activity? Almost never 17% 3x/week 15% Less than 1x/week 13% 4x/week 15% 1x/week 12% >4x/week 13% 2x/week 15%

9. Question wording Will people misinterpret the question? Does the question include assumptions or need a time frame specified? Does the question fit the population you’re sampling? How personal is the wording? Is the wording too direct or not direct enough? Are there words people wouldn’t know? Are the alternatives clear? Is the wording unbiased?

10. When did you move to CF? in 2005 When I graduated from college When I was 18

11. Responses desired must fit what P can answer In the past 30 days, were you able to climb the stairs with no difficulty? On days when you drink alcohol, how many drinks do you usually have? How many miles are you from the nearest hospital?

12. Survey wording • Don't you agree that campus parking is a problem? • There are many people who believe that campus parking is a problem. Are you one of them? • Do you agree that campus parking is a problem and that the administration should be working diligently on a solution? • What do you think about parking?

13. n More on question wording (1 = strongly disagree to 7 = strongly agree; * = reverse scored)

14. More on question wording (1 = strongly disagree to 7 = strongly agree; * = reverse scored)

15. More on question wording (1 = strongly disagree to 7 = strongly agree; * = reverse scored)

16. USA Today/Gallup Poll Feb. 20091013 adults, +/- 3% error • Do you approve of the government temporarily taking over major banks in danger of failing? 54% approved • Do you approve of the government temporarily nationalizing major banks in danger of failing? 57% disapproved

17. Another polling example • In any health care proposal, how important do you feel it is to give people a choice of both a public plan administered by the federal government and a private plan for their health insurance--extremely important, quite important, not that important, or not at all important? • 77% extremely or quite important • Would you favor or oppose creating a public health care plan administered by the federal government that would compete directly with private health insurance companies? • 43% favor

18. What bad examples did you find? • opinioncenter.com • 1.     If a bottle of water costs you three times more, would you continue buying it? • 2.     If not, what would you do? • 3.     Are you aware of the steps use to process a generic bottle of mineral water? • 4.     Do you believe that bottled water can be more expensive than oil? • 5.     When travelling overseas, do you find it easy to find your preferred bottled water brands? • 6.     How much importance do you give to drinking water on daily basis? • 7.     How concerned are you that there may be water problems, including water shortages, around the world? • 8.     Please specify any global water issues or concerns you are aware of? • 9.     Have you taken any steps to help alleviate any water problems around the world? • “The student makes the statement that Darwin killed religion”.  • “The student makes the statement that finding out whether religion is real is not a big deal in their life”.   • ‘The Store’ represents good value for the money.”

19. Question placement • What should be early vs. late in the survey? • What else affects placement?

20. Be nice to participants • Thank them to start • Keep it short • Be alert to discomfort • Thank them at the end • Send a copy of the results if they want them

21. Interviews • What is the role of the interviewer? • How should they be trained? • Should they be “blind”? • Any examples of good/bad interviewers? • What should an interviewer bring/do? • How do you decide which person to interview?

22. Parts of the interview • What are good techniques for • Getting entry • Introduction • Explanation of study • Asking questions • Probing for more information • Recording the interview • Ending the interview

23. Advantages/disadvantages and when to use: • Mail survey • Group-administered questionnaire • Household drop-off survey • Electronic survey • Focus group • Telephone interview • Face-to-face interview • CATI • CAPI • ACASI • IVR • Knowledge networks

24. Web surveys • Should you have one or multiple pages? • Should you have a line telling people how far along they are? • What other things may affect responses? • Can you get representative samples?

25. Where are we now? • Volunteer panels • Big data • Big science • Exploratory, non-theory driven is okay • Data visualization

26. Asides on the good and bad • Bad • Small n • p values • p hacking • Good • Badges • Taxonomies of situations

27. General survey issues • What are advantages and disadvantages of surveys overall? • How can the disadvantages be dealt with? • How have cell phones affected surveys? • How can you deal with unlisted numbers? • What are issues with RDD? • Mitofsky-Waksberg method • List-assisted

28. Response rates • Why are response rates decreasing? • What are reasons for nonresponse and how can you deal with them? • Call back rates? • How can survey researchers increase rates? • Are lower response rates a problem? When?

29. Issues in using different modes: • Population: • Do you have a sampling frame? • Are they able to read and/or speak English? • Will the population cooperate? • Are there geographic issues? • Sampling: • What do you know about your p’s? • Can you find them? • Who is the p? • How is your sampling frame? • How much does the response rate matter for your study?

30. Question: • What types of questions (detail, easy to understand)? • How complex are the questions (filters, etc.)? • Do you need to control order? • Are there long questions? • What are the response formats? • Content: • Will they know about the issue? • Will they need to look things up? • Are they sensitive questions?

31. Bias: • Is there an issue with social desirability? • What about interviewer effects? • Is it important to make sure the respondent is actually the respondent? • Administrative issues: • How much money do you have? • Facilities? • Time? • Personnel? • Example: http://fivethirtyeight.blogs.nytimes.com/2012/11/10/which-polls-fared-best-and-worst-in-the-2012-presidential-race/

32. Things to consider in evaluating survey data • Who paid for the survey • Who are the participants • What was the sampling frame and method • How were the questions worded • What is the margin of error

33. Missing data • When is missing data likely to be a problem? • What are the different types of missing data you can have? • MCAR vs. MAR vs. MNAR • In reality, a continuum between MAR and MNAR • What problems do they cause? • What are the “old” methods and when are they okay? • Listwise deletion • Pairwise deletion • Mean substitution • Missingness dummy variable • Regression-based single imputation

34. Modern methods • Sidebar: What is an eigenvalue and where would you find one? • EM algorithm (expectation maximization) • Goes through values one at a time. If there is a value, it’s added to the model. If not, then the best guess based on predicting it in regression with all the other variables is put in. This continues until it becomes stable. • Good for: mean, variance, covariance estimates, correlation matrices, coefficient alpha, exploratory FA • No standard errors produced, so not good for hypo testing • Use SAS, NORM, EMCOV, maybe SPSS

35. MI (multiple imputation): • Better because it doesn’t assume that responses lie on the regression line—it adds in random error • Use NORM, SAS, Splus • PAN program—uses growth curves or clustered data • FIML: • Does it all in one step • SEM software: Amos, LISREL, Mx

36. Is it okay to “make up data”? • Inclusive variable inclusion strategies • Include variables that are correlated, even if not in model • Reduces bias • Helps with power • How to deal with missing scale items: • If just part of scale, can use partial data if at least ½ variables and good alpha and all item-total correlations are about the same • Otherwise, impute at the individual level

37. Issues with MI: • Works even with n = 50, 18 predictors and 50% missing • Works with nonnormal data • Don’t round • Use lots of imputations (e.g., 40 for 50% missing) • Consider whether to impute separately for groups (e.g, gender) • Can use with clustered data (dummy code) • Can use with categorical data • Can use with lots of variables

38. Planning for missing data • 3 form design • All get some questions, others just some get • 2 method measurement • Get data from everyone on cheap measure and from a sample on “expensive” measure • Include good predictors of the missing values in your data set • Measure p’s plans to drop out • Follow up and try to get measures for some drop outs • Why isn’t looking at difference between stayers and leavers a cure?

39. Diary methods • Event sampling, ESM • Rochester Interaction Record • More naturalistic • Multi-level

40. Social network analysis • Longer history in sociology • Sociograms, centrality • Clustering, social influence, selection

41. Next week… • Experimental design and mediators/moderators • Make sure you understand the difference between them and how to test for them