Designing Questions for the Standardized Survey

1. Designing Questions for the Standardized Survey Professional Development September 19, 2008

2. Goals of the workshop • Present the theoretical foundation for effective question design • Review the principles of good questions • Provide an overview of examples of PRA practice from which staff may draw

3. Outline • What is a standardized survey? • Questions 101 • Framing questions and cognitive interviewing • Some advanced issues • Semantic differential • Likert scales • Magnitude scales • Some recurrent issues • middle position • changing the questionnaire mid-survey • sensitivity to mode • Accommodating uncertainty — split ballots and experimentation

4. Standardized surveys • A standardized question presents only fixed categories as potential responses or asks for a specific “code-able” response: • What is you age? • I am going to read you some age ranges. When I come to the one that applies to you, please stop me. • Please check the age range that applies to you. • At times, respondents can be asked to offer an opinion that is recorded by interviewers in a notes field or by the respondent in a special area of the question. • Do you like or dislike going to the centre? <Interviewer records answer in CATI> Why is that? <Interviewer records verbatim in a notes field>

5. Qualitative elements in a standardized questionnaire • Verbatim responses may be • coded into fixed categories (create a new variable) • be reviewed as “qualitative data” • ignored (included solely to convince the respondent that we care about his/her opinion • Minimize the use of verbatim or open-ended questions in standardized questionnaires. • Caution: There are two meanings of “qualitative.” • 1. It can refer to verbatim texts, video, audio, or some other visual/auditory content. Qualitative coding reduces complex information to a simple numerical category. • The star system and thumbs up/down for movies are examples of extremely reductive codes. • In statistical analysis, qualitative data indicate a change in state and are also referred to as “dummy” variables.

6. Questions 101 – The basic structure

7. Question phrasing ‘rules’ • Set wording to the respondent — Make sure that jargon and acronyms will be understood by the respondent. • Use a Flesch-Kincaid Grade 8 level for the general population and Grade 11 for civil servants and professionals. • Use technical jargon with specialized audiences to communicate that you understand issues. • Short sentences in the active voice work best. • Balance alternatives • “Some people support sending Canadian troops to Afghanistan; others do not. What do you think?” • Instead of “What do you think of sending Canadian troops to Afghanistan?”

8. Question phrasing ‘rules’ (cont’d) • Avoiding leading the respondent • “In order to balance the budget, should government reduce spending on left-wing artists or raise taxes on socialists?” is clearly a biased question. • Instead, use two questions: • Doyou agree or disagree that the budget should be balanced? • [If agree] Should government reduce spending on left-wing artists or raise taxes on socialists? • Some people support increasing taxes on socialists, while others support reducing spending on left-wing artists in order to balance the budget. What do you think? Or better still, use the same phrasing as the previous slide for the second question: For stylistic reasons, one might be tempted to replace the second use of “support” with a word like “advocate,” but this makes the question unbalanced since “advocacy” is a stronger word than “support”.

9. Question phrasing ‘rules’ (cont’d) • Ground and focus the question (See section on cognitive interviewing) • Avoid hypothetical questions that are improperly framed. Special techniques (discrete choice and conjoint) create a structure for these questions (see below). • Don’t know, no opinion, neutral, and not applicable are different • Don’t know – respondent is unfamiliar with the topic and cannot form an opinion • No opinion – respondent knows the topic but is disinterested in any alternative • Neutral – respondent knows the issue, but has adopted a position within the extremes • Not applicable – respondent is not eligible to respond, regardless of their knowledge or strength of position.

10. Framing questions and cognitive interviewing Classic study (Belson, 1981) In a face-to-face interview, respondents were asked to agree to disagree with a series of statements such as “television shows are too violent for children.” After carefully recording the responses, respondents were approached the day after and “debriefed” about the survey. Interviewers asked the respondents what they meant by “television shows,” “too violent,” and “children”. Belson discovered that these terms meant different things to different people, and he detected distinct meanings. • Television show meant prime time to some, and all times to others. • Children meant under 6, under 12, and under 18 depending on the respondent. • Too violent had meaning specific for each individual. With three meanings of child and two meanings of TV show, there are six questions being asked, let alone the infinite shades of “too violent.”

11. If all the problems of question wording could be traced to a single source, their common origin would probably prove to be in taking too much for granted. S. Payne, The Art of Asking Questions, 1951

12. Cognitive interviews • The fundamental challenge in a standardized interview is to phrase the question in a way that everyone will understand what the research means. • This requires extensive pretesting using cognitive interviewing • Cognitive interviewing uses “think-alouds” and “probes.” • Think-alouds allow the respondent to collect their thoughts verbally “In the last six months have you been to the dentist” <yes, let me see, I had my teeth cleaned six months ago and last week has to go for a repair on a chipped tooth.. Oh yes, I had a tooth ache last April> • Probes direct specific questions “Tell me about the last time you went to the dentist.” “Was this for a checkup or to deal with a problem?” “Why are you certain that this last visit was in June?”

13. Grounding the question • Cognitive interviewing assesses response variation in a specific question. • It is a pretest technique for mapping response variation and error (deviation from the intended meaning). • Two models: • Ericsson-Simon model (1980) rests on the presumption that people can remember why they responded to a question the way they did. Experimental evidence shows that this works when the recall task involves verbal information (as opposed to non-verbal/spatial information), is novel, endures for a period of time, and has happened recently. It also works when subjects are asked to describe “what” as opposed to “why” they did something. • Task analysis (Tourangeau, 1984). The theory behind task analysis is that questions are processed according to a protocol and answers are provided within a specific values context.

14. Task analysis contextsQuestion-answer processing • Cannell et al. (1981) • comprehension • decision/retrieval/organization of data • response evaluation (filtering) • response output • Martin (1983) • giving meaning to the question • searching for relevant data • formulating a judgment • Tourangeau (1984) • comprehension • retrieval • judgement • response

15. Framing • Many questionnaires make excessive demands on memory. • The term “recall bias” is misleading or oversimplified when it actually means “collecting really bad data.” • Framing practices • Use introductions and questions to set the stage. • Send a letter in advance explaining the survey and reminding the respondent of key dates (e.g., Our records show you were a patient at Acme Cardiac and Rotor Rooter Unit four months before you died.) • Avoid asking detailed questions about events or states in the past.

16. Advanced issues: response scales • Attributes of a good response scale • Respondents and researchers need to accept that respondents’ feelings, perceptions, and judgements can be described by numerical, semantic, and physical analogues. • Respondents must share the same interpretation of the scale. • A scale must discriminate among levels. • A scale value must mean something. “Very good” must mean something different (better) than “average”. Example: (telephone or mail) On a scale of 0 to 10, where 0 is poor and 10 is excellent, what is your overall opinion of the program? Example: (mail) Using the scale below, please rate your last experience with the emergency room. -2 -1 0 1 2 (Very Negative) (Very Positive)

17. Magnitude scales • Advantages • Identify a middle position (0 – 10) • Linear with even steps • Supports statistical measures of central tendency and variance • Disadvantages • Need to translate subjective concepts into a number • Linear scales may not capture intensity at the extremes • Easy to overuse • Automatic massing at the mid-point Example: (telephone or mail) On a scale of 0 to 10, where 0 is poor and 10 is excellent, what is your overall opinion of the program? Example: (mail) Using the scale below, please rate your last experience with the emergency room. -2 -1 0 1 2 (Very Negative) (Very Positive)

18. Two common biases Qa Contaminates Qb • Inter-item contamination Qa. In your view, is AIDS a threat to someone who is heterosexual and not a drug user? Qb. Is the government providing sufficient funding to basic research in health? • Social desirability bias Qc. Have you heard of the XYZ program? Qd. In order to assess how well we are promoting our services, please tell me whether you have heard of the XYZ program. Challenges Shifts blame and allows someone to admit ignorance

19. Negative framing Scenario 1 Imagine that Canada is preparing for the outbreak of an unusual disease that is expected to kill 600 people. Two alternative programs have been proposed with the following outcomes: [respondent preferences in brackets] Program A 200 people will certainly be saved. [72 favoured this one] Program B There is a 33% chance that 600 will be saved and a 67% chance that no one will be saved. [28 favoured this one] Scenario 2 Same basic scenario as above, but with the following two program alternatives: Program C 400 people will certainly die. [22 favoured this] Program D There is a 33% chance of 0 deaths and a 67% chance of 600 deaths. [78 favoured this] Negative framing affects response Framing is neither good nor bad, but a feature of a linear structure sequence to a conversation.

20. Plain language • http://s50.photobucket.com/albums/f341/KMS102177/Videos/?action=view&current=GeorgeCarlinonLanguage.flv

21. CasesLMAPDAgri-environmental programming