Designing the Questionnaire or Observation Form

1. Designing the Questionnaire or Observation Form

2. Procedure for Developing a Questionnaire Step 1 Specify what information will be sought Step 2 Determine type of questionnaire and method of administration Step 3 Determine content of individual questions Step 4 Determine form of response to each question Step 5 Determine wording of each question Step 6 Determine question sequence Step 7 Determine physical characteristics of questionnaire Reexamine steps 1-7 and revise if necessary Step 8 Step 9 Pretest questionnaire and revise if necessary

3. Open-Ended Questions • How old are you? ______ • How do you feel when you drive your Toyota minivan? Why do you feel that way? • Can you name three sponsors of the Monday night football games? _____ • Do you intend to purchase an automobile this year? _____ • Why did you purchase a Sony brand color television set? _____ • Do you own a VCR? _____

4. Multichotomous Questions • A fixed-alternative question in which respondents are asked to choose the alternative that most closely corresponds to their position on the subject. • Example • How old are you? • ____ Less than 20 • ____ 20-29 • ____ 30-39 • ____ 40-49 • ____ 50-59 • ____ 60 or over

5. Dichotomous Questions • A fixed-alternative question in which respondents are asked to indicate which of the two alternative responses most closely corresponds to their position on the subject. • Example • Do you think laws requiring passengers in motor vehicles to wear seat belts are needed? • Yes • No

6. Split Ballot A technique used to combat response bias in which one phrasing is used for a question in one-half of the questionnaires while an alternate phrasing is used in the other one-half of the questionnaires. Example Do you think gasoline will be more expensive or less expensive next year than it is now? More expensive Less expensive Do you think gasoline will be less expensive or more expensive next year than it is now? More expensive Less expensive

7. Item Non-Response A source of non-sampling error that arises when a respondent agrees to an interview but refuses, or is unable, to answer specific questions.

8. Flame-broiled or Fried? Original Question: “Do you prefer your hamburgers flame-broiled or fried?” Result: flame-broiling (Burger King) beat frying (McDonald’s) by a 3-1 margin Revised Question #1: “Do you prefer a hamburger that is grilled on a hot stainless-steel grill or cooked by passing the raw meat through an open gas flame?” Result: 53% preferred the grill (McDonald’s) Revised Question #2: “The chain that grills on a hot stainless-steel griddle serves its cooked hamburgers at the proper temperature without having to use a microwave oven. And the chain that uses the gas flame puts the hamburgers after they are cooked into a microwave oven before serving them. Just knowing this, from which of these two chains would you prefer to buy a hamburger?” Result: the grill (McDonald’s) beat the flame (Burger King) by a 5.5 to 1 margin

9. Leading Questions • A question framed so as to give the respondent a clue as to how he or she should answer. • Example • Do you feel that limiting taxes by law is an effective way to stop the government from picking your pocket every payday? • Yes • No • Undecided

10. Implicit Alternative A problem that occurs when an important alternative or response option is not included in a question (or response categories). Example Would you like to have a job, if this were possible? Yes No Would you prefer to have a job, or do you prefer to do just your housework? Yes No

11. Implicit Assumption A problem that occurs when a question is not framed so as to explicitly state the consequences, and thus it elicits different responses from individuals who assume different consequences. Example Would you like to double the number of job offers you receive as a senior? Yes No Would you like to double the number of job offers you receive as a senior if that means devoting an additional 10 hours per week to studying so as to raise your grade point? Yes No

12. Example Are the food and service at the local Pizza Hut good? Yes Double-Barreled Question A question that calls for two responses and thereby creates confusion for the respondent. No Key: Be on the lookout for “and” and “or” in your items!

13. Guidelines for Question Wording • Use simple words and questions • Avoid ambiguous words and questions • Avoid leading questions • Avoid implicit alternatives • Avoid implicit assumptions • Avoid generalizations and estimates • Avoid double-barreled questions

14. Guidelines for Question Sequencing • Use simple, interesting opening questions • Use the funnel approach, asking broadquestions first • Carefully design branching questions • Ask for classification information last • Place difficult or sensitive questions near the end

15. Guidelines to Question Sequencing (Cont) Funnel ApproachAn approach to question sequencing that gets its name from its shape, starting with broad questions and progressively narrowing down the scope. Example How would you rate your last experience when eating at ABC restaurant? Extremely Extremely Poor Poor So-So Good Good How would you rate the quality of the service? Extremely Extremely Poor Poor So-So Good Good

16. Example of Question Order Bias • Demand estimation for new product • Key question: buying interest (5-point scale, “not at all interested-very interested”) Questions asked before buying % very interest question interested None 3 Asked about advantages 17 Asked about disadvantages 0 Asked about advantages and disadvantages 6

17. Guidelines to Question Sequencing (Cont) Branching Question A technique used to direct respondents to different places in a questionnaire, based on their response to the question at hand.

18. Guidelines to Question Sequencing (Cont) • Branching QuestionExample • 1. Please answer the following question Yes or No. Did you, personally, give money to Canopy of Care last year? Y or N • {If Yes, go to #3.} • We are interested in why people do not contribute. The following is a list of answers others have given. Please tell me which, if any, apply to you. • a. ___ Someone else in my household had already contributed. • b. ___ I did not have the money at the time. • c. ___ I gave to other charities. • d. ___ I volunteered my services to Canopy of Care instead of contributing money. • e. ___ I volunteered my services to other charities instead of contributing to Canopy of Care. • f. ___ I did not give because Canopy of Care spends its money inefficiently. • g. ___ None of the above. • To how many different charities do you think Canopy of Care gives money? • a. ___ 0-20 d. 81-100 • b. ___ 21-40 e. More than 100 • c. ___ 41-80 f. Don’t know.

19. Pretest Use of a questionnaire (or observation form) on a trial basis in a small pilot study to determine how well the questionnaire (or observation form) works. ALWAYS PRETEST DATA COLLECTION FORMS!!!!!

20. Handling Sensitive Questions • Don’t ask unless absolutely necessary! • Give broad response categories (e.g., income, age) • Place near end of questionnaire • “Hide” them in less sensitive questions • Use counterbiasing statement • Use randomized response technique

21. Use of Counterbiasing Statement • Recent studies have shown that a high percentage of males use their wives’ cosmetics to hide blemishes. Have you used your wife’s cosmetics in the past week?

22. Randomized Response Model An interviewing technique in which potentially embarrassing and relatively innocuous questions are paired, and the question the respondent answers is randomly determined but is unknown to the interviewer.

23. 160 “yes” responses 240 “no” responses Randomized Response Technique (example from Weiers 1988) Initial, “benchmark” study with nonsensitive question: “Do you likepepperoni pizza?” (65% say “yes.”) Second survey, involving 400 respondents. Each respondent flips coin, with flip result not revealed to researcher. Heads Tails Respondent answers question 1: “Do you like pepperoni pizza?” Respondent answers question 2: “Have you ever been in jail?”

24. 160 “yes” responses 240 “no” responses Randomized Response Technique (example from Weiers 1988) “Yes” “No” 130 (.65x200) 70 200 (.5x400) Pizza Jail 30 (160-30) 170 200 160 240 Conclusion: Estimate that 30/200, or 15% of population has “been in jail.”

25. Physical Characteristics of Questionnaire: Suggestions • questionnaire should appear simple to complete (white space is your friend!) • minimize number of pages (smaller fonts are OK, provided form appears simple) • mix-up response formats occasionally (avoids response set bias and breaks monotony) • use directions as necessary for each group of items, but keep them short and simple • number items within each section • have respondents check boxes rather than lines • use shading, boxes, lines, etc., to keep it interesting

26. Other Issues • Pretest! • Know exactly how each piece of information will be used • NO “nice-to-know” questions! • Back translation for international questionnaires • Pre-coding