Survey Research

1. Survey Research Prying into Your Personal Life in Ways Telemarketers Could Never Dream Of

2. Appropriate Research Questions for Surveys • Self-reported beliefs and behaviors • Ask things at one time, measure many variables, and test several hypotheses

3. Appropriate Research Questions for Surveys • Behavior • How often do you brush your teeth? • Attitudes/Beliefs/Opinions • What is the biggest problem facing the nation today? • Characteristics • Do you belong to a union? • Expectations • Do you plan to buy a new car in the next year? • Self-classification • Are you liberal or conservative? • Knowledge • Who was elected mayor in the last election?

4. Key Limitation • Surveys ONLY provide data on what a person or organization says • This can differ from what they actually do or think • Survey research is DEDUCTIVE

5. Steps in Conducting a Survey • Step 1: • Develop hypotheses • Decide type of survey • Write survey questions • Decide on response categories • Design layout

6. Steps in Conducting a Survey • Step 2: • Plan how to record data • Pilot test survey instrument • Step 3: • Decide on target population • Get sampling frame • Decide on sample size • Select sample

7. Steps in Conducting a Survey • Step 4: • Locate respondents • Conduct interviews • Carefully record data • Step 5: • Enter data into computers • Recheck all data • Perform statistical analysis on data

8. Steps in Conducting a Survey • Step 6: • Describe methods and findings in research report • Present findings to others for critique and evaluation

9. Constructing the Questionnaire • Introductory remarks and instructions • Keep it clear, keep it simple, keep respondent’s perspective in mind • Question writing is more of an art than a science • Skill, practice, patient, and creativity

10. Questionnaire Hints • 1) Avoid jargon, slang, and abbreviations • No abbreviations • Only use this if you’re surveying a specialized population • “Did you drown in brew until you were totally blasted last night?”

11. Questionnaire Hints • 2) Avoid ambiguity, confusion, and vagueness • “What is your income?” • Confusion causes inconsistencies • 3) Avoid emotional language • Implicit connotative as well as explicit denotative • Use neutral language • “What do you think about a policy to pay murderous terrorists who threaten to steal the freedoms of peace-loving people?”

12. Questionnaire Hints • 4) Avoid prestige bias • Avoid associating statements with them • “Most doctors say that cigarette smoke causes lung disease for those near a smoker. Do you agree?” • 5) Avoid double-barreled questions • Each question = ONE QUESTION ONLY • “Does this company have pension and health insurance benefitted?”

13. Questionnaire Hints • 6) Do not confuse beliefs with reality • Don’t mix up respondents beliefs with what we want to measure • “Do you rate a teacher higher if the teacher tells many jokes?” • 7) Avoid leading questions • Don’t lead them to an answer • “You don’t smoke, do you?”

14. Questionnaire Hints • 8) Avoid asking questions that are beyond respondents’ capabilities • Frustrates respondents • Produces poor-quality responses • “Two years ago, how many hours did you watch TV every month?”

15. Questionnaire Hints • 9) Avoid false premises • Can’t put a premise that respondent could disagree with without risking alienating them • “When did you stop beating your girl/boyfriend?”

16. Questionnaire Hints • 10) Avoid asking about intentions in the distant future • Hypotheticals are bad • Not specific or concrete • “After you graduate college and have kids, will you invest in the stock market?”

17. Questionnaire Hints • 11) Avoid double negatives • Grammatically incorrect and confusing • “Do you disagree with those who do not want to build a new city swimming pool?”

18. Questionnaire Hints • 12) Avoid overlapping or unbalanced response categories • Mutually exclusive: categories do not overlap • Exhaustive: every respondent has a choice • Balanced: Offer polar opposites at each end of a continuum • “Did you find the service at our hotel to be, Outstanding, Excellent, Superior, or Good?”

19. Aiding Recall • Ability to recall accurately declines over time • Memory is less trustworthy than once assumed • Give respondents special instructions and extra thinking time to guide them

20. Types of Questions • Threatening Questions • Socially Desirable Questions • Knowledge Questions • Skip or Contingency Questions

21. Threatening • Sensitive issues…people don’t want to talk about them • Underreport socially undesirable answers • Context and wording are how to fix this • Issue examples: • Masturbation, sexual intercourse, illicit drug use, income

22. Socially Desirable • Over report socially desirable answers (Social desirability bias) • Giving money to charity, voting etc. • Have to make the norm violation seem less objectionable then it really could be

23. Knowledge • We can’t answer elementary questions as Americans… Are You Smarter than a 5th Grader?? • Pilot-test to make sure they’re at an appropriate level of difficulty

24. Skip or Contingency • Two or more part question • Answer to one part determines the next question you’ll receive • Sometimes called screening questions

25. Open v. Closed Questions • Open-ended: Respondent can give any answer they want • Close-ended: Gives respondents fixed answers from which to choose • Big surveys use close-ended…quicker and easier • Can mix both methods • Open-ended best in early or exploratory stages of research • Partially open questions leave final answer choice • as other

26. Advantages of Closed • Easy and quick to do • Easy to compare • Easier to code and analyze • Response choices can clarify meaning of question • Replication is easier

27. Disadvantages of Closed • Suggest ideas that respondent may not have thought of • No opinion people can answer anyways • Frustrating if desired answer is not a choice • Simplistic responses to complex ?s • Force us to make choices we wouldn’t make in the real world

28. Advantages of Open • Unlimited responses • Answer in detail and clarify • Unanticipated feelings can be discovered • Creativity and self-expression • Reveal logic

29. Disadvantages of Open • Different degrees of detail in answers • Can be irrelevant • Hard to compare/analyze • Coding is tough • Hard to write verbatim in interviews • More time necessary • Take up lots of space on questionnaire

30. Nonattitudes and Middle Positions • Debate whether to include choices for middle, neutral, or nonattitudes • Two errors: • Accept nonattitude when respondent has one • Force choice when respondent doesn’t have one • Standard-format v. quasi-format v. full-format

31. Agree/Disagree, Rankings, or Ratings? • Agree/Disagree • Rankings • Ratings • Best to offer respondents explicit alternatives • Avoid the response set bias (just check agree to everything) • Respondents can rate several items equally high, but will put them in hierarchy if asked to

32. Word Issues • Need to use simple vocabulary and grammar to minimize confusion • Need to watch for effects of specific words or phrases • Forbid v. not allow • Certain words trigger an emotional reaction • “To help the poor” v. “for welfare”

33. Length of Survey • Researchers prefer long because they’re cost effective • Phone: 5-20 minutes • Mail questionnaire: 3-4 pages • Face-to-face: Around an hour

34. Order Effects • Strongest for people who lack strong views, less educated, elderly • May not perceive each element of a survey as isolated and separate • Influence through content and response

35. Context Effects • Funnel sequence: • Start general and get more specific • Split sample • Half get questions in one order, other half different • More ambiguous the question, the more context effects matter

36. Survey Layout • Clear, neat, and easy to follow • Number questions • Identifying info • Cover sheet • Professional appearance • Provide instructions

37. Survey Layout • Crucial in mail questionnaires • Use letterhead • Always thank them for participation • Circle response v. check box v. fill in dots v. put an X in a blank • List answer categories down, not across

38. Survey Layout • Use arrows and instructions for contingency questions • Matrix questions • Compact way to present a series of questions with the same response categories • Box 7.5

39. Nonresponse • If not enough people respond, we may not be able to generalize • Many people are burnt out with surveys these days • Improve eligibility rates through careful respondent selection

40. Nonresponse • Decrease refusals through sending letters before calling, small incentives, adjusting behavior, using alternative interviewers, using alternative interview methods • Initial contact is HUGE in determining likelihood for success

41. Surveys • Survey research is research based on direct or indirect interview methods. • Many types of surveys: • In person (including interviews) • Mail • Telephone • E-mail and Internet • Group surveys

42. Mail and Self-Administered Questionnaires • Advantages: • Cheaper • Wide geographical area • Completed when convenient • Respondent can check personal records • Anonymity

43. Mail and Self-Administered Questionnaires • Disadvantages • Low response rate • Can’t control conditions under which its filled out • Someone other than intended can fill out • Can’t observe reactions • Ill suited for those who don’t have good grasp of English

44. Mail and Self-Administered Questionnaires • How to Increase Response • Address to specific person • Careful cover letter on letterhead • Postage-paid, addressed return envelope • Neat, attractive layout • Send two follow-up reminders • Nothing during major holidays • No questions on back page • Include a small monetary inducement

45. Web Surveys • Advantages • Very fast • Inexpensive • Flexible design • Can use visuals or audio

46. Web Surveys • Disadvantages • Coverage (75% have internet) • Multiple e-mail addresses • Self-selection • Protecting privacy and confidentiality • Encrypt • Complexity of design depends on software • Screen-by-screen questions • Progress indicator • Needs to be easy to move back and forth across questions

47. Phone Interviews • Advantages • 95% coverage rate • Can interview about 1500 in a few days • Half the cost of face-to-face interviews • CATI

48. Phone Interviews • Disadvantages: • Higher cost • Limited interview length • Use of interviewer reduces anonymity and introduces potential interviewer bias

49. Face-to-Face • Advantages: • Highest response rates • Permit the longest questionnaires • Can observe surroundings • Ask all types of questions

50. Face-to-Face • Disadvantages: • High cost • Training, travel, supervision, personnel costs • Interviewer bias