IEOR 170: Questionnaire Design and Paper Prototyping

1. IEOR 170: Questionnaire Design and Paper Prototyping Jingtao Wang 2/21/2007

2. Administrivia No Confliction Between IEOR 170 and IEOR 171 There is no planned final exam for IEOR 171 up till now. Please contact Dina Michail dina@ieor if you can not enroll these two classes at the same time More than three students are still on the waiting list. Get yourself enrolled if you have completed all the assignments up till now.

3. Administrivia Group assignment can not be delayed. Every member will receive a zero if not submitted in time Two peer-reviews about your team members for this semester The Contextual Inquiry Assignment is out today. Due in two weeks Get an early start!

4. Previously on IEOR 170 Task Analysis Eleven Task Analysis Questions Contextual Inquiry Why Context Inquiry? Major Principles What is Persona?

5. Previously on IEOR 170 The Needfinding process Takeout box design practicing

6. Topics Questionnaire Design Paper Prototypes

7. What is a Questionnaire? An instrument (form) to Collect answers to questions Collect factual data � gathers information or measures A series of written questions / items in a fixed, rational order

8. Why Using a Questionnaire? A well designed questionnaire can Give appropriate data which answers your research question Minimize potential sources of bias Will more likely be completed

9. Advantages of Questionnaires Can reach a large number of people relatively easily and economically (especially postal/E-mail questionnaires) Provide quantifiable answers Quick and easy to conduct Relatively easy to analyze Get quick feedback on a range of ideas Can get person�s initial reaction to an idea Can get detailed information from a person

10. Disadvantages of Questionnaires Provides only limited insight into problem Limited response allowed by questions Maybe not the right questions are asked Varying response Misunderstanding/misinterpretation Need to get it right first time Hard to chase after missing data Often takes place away from natural setting Recall problems High probability of False positives: user thought something would be an issue but it wasn�t Missed problems: user didn�t catch an issue

11. Common Pitfalls In Questionnaire Design Basic tenet: Ask a question, and you�ll get a response! Translation: Garbage in, garbage out!

12. Lessons of Consumer Survey When Levi Strauss & Co. asked students which clothes would be most popular this year, 90% said Levi�s 501 jeans. (They were the only jeans on the list.)

13. Why Xbox was Not Successful in Japan ? Only around half million units were sold in Japan in four years Survey Results from Japan Too Big (rooms in Japan are small) Too Heavy Not Stylish Few RPG games

14. Why Xbox 360 Again Couldn�t Change the Game in Japan ? Smaller than PS3 Less Heavy than PS3 As �Stylish� as PS3 More RPG games than PS3 in Japan at this time

15. Types of Questionnaire Self-administrated By post E-mail/Internet Interviewer-administrated Face to face Telephone

16. Self-administered Questionnaire Advantages Cheap and easy to administer Preserves confidentiality Completed at respondent's convenience Administered in a standard manner No influence by interviewer

17. Self-administered Questionnaire Disadvantages Low response rate Questions can be misunderstood No control by interviewer Time loss

18. Interview-administered Questionnaire Advantages Participation by illiterate people Clarification of ambiguity Quick answers

19. Interview-administered Questionnaire Disadvantages Interviewer bias Needs more resources Only short questionnaires possible Difficult for sensible issues

20. Objectives of a Questionnaire To ensure that obtained information is relevant, accurate and generalizable to the study i.e. Validity, Reliability and Representiveness

21. Format of Questions Two main question formats Closed format ? give a �fixed� response Yes, No, Don�t know Always, sometimes, never Open format ? allow people to express their views in their own words: What is your most distressing symptom? Please describe: ____________________________________________________________________________________

22. Open or Closed? Closed � forced choice Advantages Simple and quick Reduces discrimination against less literate Easy to code, record, analyse Easy to compare Easy to report results Disadvantages Restricted number of possible answers Loss of information Possible compromise Insert field �others�

23. Open or Closed? Open format � free text Advantages Not directive Allows exploration of issues - generate hypothesis Used even if no comprehensive range of alternative choices Good for asking about knowledge and attitudes Detailed and unexpected answers possible Disadvantages Answer depends on interviewer Time-consuming Coding problems Difficult to analyse! Difficult to compare groups

24. Closed Questions Straightforward response What is your age in years? ___ years How long have you stayed in Hotel X? ___ days What is your sex (gender)? Male ? Female ? Did you stay in Hotel X on 23/7/04? Yes ? No ? Don�t know ? Did you eat dinner in restaurant x on 23/7/04? Yes ? No ? Don�t know ?

25. 2. Checklist Which of the following did you eat for dinner on 23/7/04? Chicken ? Beef ? Ham ? Salad ? Egg mayonnaise ? Closed Questions

26. Closed Questions 3. Rating scale Did you use condoms during the past six months? With� Always Sometimes Seldomly Never Steady partner ? ? ? ? Casual partners ? ? ? ? Sex workers ? ? ? ?

27. Closed Questions 4. Numerical rating scale How useful would you think that information on hepatitis A from the travel agency would be? (please circle) 1 2 3 4 5 6 7 Not at all useful Very useful

28. Closed Questions 5. Scales for measuring attitude (Lickert) Travel companies should not offer anymore hotels linked to hepatitis A No, I strongly disagree ? No, I disagree quite a lot ? No, I disagree just a little ? I�m not sure about this ? Yes, I agree just a little ? Yes, I agree quite a lot ? Yes, I strongly agree ?

29. Open Questions Often used in qualitative research, focus groups, in trawling questionnaires What do you think is the reason you became ill? ____________________________________ Do you think service X would make your life easier? If so, in what way? If not, why not? ________________________________________________________________________________________________

30. Basic Principles - Question Order Easy ? difficult General ? particular Factual ? abstract Starting questions Simple With closed format Relevant to main subject Non-threatening Neither demographic nor personal questions Be aware of ordering effects!

31. Basic Principles - Question Order 2 Group questions by Topic/ response options Don�t put most important item last Questionnaire likely to be completed if Relevant Logical

32. Basic Principles � Question Format Ask for one information at a time: Did you go swimming in the pool and have cocktails at the bar during night or day? Yes � No � Remember option �don�t know� Mutually exclusive and exhaustive answer options Vertical order of answer options

33. Basic Principles � Be Precise Did you swim often in the pool? Yes � No � vs. How often did you swim in the pool? Once � Twice � Three times or more � Not at all � Don�t know �

35. Basic Principles � Be Appropriate Are you a drunk? Yes � No � vs. How often did you consume alcoholic beverages during your holiday? Daily � 2-6 times/week � Once a week � Less than once a week � Don�t know �

36. How to ask a sensitive question(Barton, POQ 1958) The casual approach: "Do you happen to have killed your partner?� The numbered card approach: "Would you please read off the number on this card which corresponds to what became of your partner? 1 Natural death 2 I killed her/ him 3 Other"

37. The everybody approach: (also called counterbiasing method) "As you know, many people have been killing their partners these days. Do you happen to have killed yours?" The other people approach: "Do you know any people who have murdered their partners? How about yourself?"

38. Personal Information - Build rapport, legitimize Embarrassing Information (downward bias) counterbiasing statement: �Recent studies indicate that a man often uses his wife�s hair spray. Do you use your wife�s hair spray?� projective techniques randomized response techniques Prestige, Normative Information (upward bias) Sensitive Questions(continued)

39. Asking Sensitive Questions: Rules of Thumb Hide in a group of more innocuous questions State that the behavior or attitude is not unusual Phrase in terms of others and how they might feel or react Use multiple-choice response categories Randomized response

40. Basic Principles - Be Objective Did you drink the strange pink drink? Yes � No � vs. Which beverage did you consume? Water � Beer � Wine � Karkad� � None of them � Don�t know �

41. Basic Principles � Be Simple & Clear Did you smoke not less than a mean amount of 7 cigarettes/2 days from 1999 onwards? Yes � No � vs. Did you smoke a mean amount of 2 pack of cigarettes/week for the last 5 years? Yes � No � Don�t know �

42. Be Simple & Clear � Common Pitfalls Avoid jargon/abbreviations/slang How often do you get up at night to PU? (pass urine) Should IVDUs be treated in the community? Avoid not mutually exclusive options What age are you? 16-20 ? ? 20-25 ? ? 25-30 ? ? 35-40 ? ?

43. Be Simple & Clear � Common Pitfalls Avoid leading questions Do you think that the food in the hotel made you sick? Did the hotel staff seem unhygenic to you? Do you agree that the hospital staff were close to exhaustion? Avoid making questionnaire too long Typographical / spelling errors

44. Examples A Gallup poll sponsored by the disposable-diaper industry asked: �It is estimated that disposable diapers account for less than 2% of the trash in today�s landfills. In contrast, beverage containers, third-class mail and yard waste to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?� (84% said no.)

45. Minimize Dichotomous Questions Dichotomous: a grammatical structure that suggests a yes or no answer Bad = �Are you satisfied with writing letters?� Good = �How do you feel about writing letters?�

46. �Why� Questions � Take Care �Why� questions are problematic because they� Presuppose things happen for a reason & the respondent knows the reason Require respondents to make analytical & deductive inferences � hard The Xbox in Japan survey

47. �Why� Questions � How to Use �Why� can give many types of responses (�Why do you want to learn email?�): programmatic (�Because it takes place at a convenient time.�) personality (�Because I�m a joiner.�) information (�Because a friend told me about it.�) social influence (�Because my priest thought it would be good for me.�) economic (�Because it was inexpensive.�) outcomes (�Because I wanted to learn about the things they�re teaching in the program.�) personal motivation (�Because God directed me to join.�) philosophical (�Because it was there.�) Decide before the interview which of these types is valuable to your goals. Word question to isolate that type Social influence example: �What other people, if any, motivate you to want to learn email?�

48. Validity, Reliability, Representiveness Validity: Asking the right question Reliability: Receiving the right response Reducing Response Error Representativeness: Generalizing results. The answers are from the right people Sample selection and sample size Reducing Non-response Error

49. Reliability Similar results by comparable measures of the same construct Two ways to measure reliability Test-Retest:: stability over time Split-Half: consistency across ratings within a multiple-item scale Reliability is easier to determine than Validity If a measure is valid, then it is reliable If it is not reliable, it can not be valid If it is reliable, it may or may not be valid

50. Introducing New Coke: Research Conducted 2000 interviews in 10 major markets: Control for brand names; Storyboards and Mock Advertisements: Try? Switch? Be Upset? Focus Groups: Mixed; loyal drinkers unfavorable. Blind Taste Tests: n = 30,000-40,000 New vs. Old Coke: 55 vs. 45; vs. Pepsi: 54-46 Generalizability: n=191,000 in 13 cities; Blind taste tests of 4 new vs. old / Pepsi.

51. New Coke: Key Research Lessons Predictive Validity Did not research impact of the Loss of Coke Construct Validity: Buy brand vs. prefer taste? Reliability: Discriminate/ Prefer

52. In-Class Group Discussion Suppose you have been recruited by Nintendo American to decide whether they should also enter the cell phone market like apple in the U.S. Try to design a market survey questionnaire (no more than 10 questions) to collect important data to answer this question What Kind of information you should collect? How to make sure your questionnaire is valid, reliable and generalizable?

53. Sources of Error in Surveys Non-response bias due to refusals Fear of consequences of participation, represents an invasion of privacy, anxiety about the subject Inaccurate responses Inability to respond: ignorance, memory problems, problems formulating an answer Unwilling to respond accurately: concerns about invasion of privacy, time pressure/fatigue, prestige seeking/social responsibility, response style/bias Error causes by interviewers Providing clues to answers Inadequate questioning or probing

54. Bias in Questions Bias = systematic differences in the measurement of a response Recall bias Cases more likely to remember than controls Observer bias Different interviewer � different interpretations Different interpretation of similar questions Reduce by structured questionnaire Non-response bias Those who respond are different from those who do not Telephone interviews: more females

55. Watch Out: Garbage In, Garbage Out Behavioral Questions: Memory Biases Omission Telescoping Attitude Questions: Confusion Double-barreled, leading, ambiguous questions Sensitive Questions: �Lying� Item non-repsonse Termination Distortion

56. Examples of Questions from Consumer Surveys A survey for Black Flag said: �A roach disk� poisons a roach slowly. The dying roach returns to the nest and after it dies is eaten by other roaches. In turn, these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches?� Not surprisingly, 79% said effective!

57. Reducing Response Errors How do respondents answer survey questions? Guess Randomly choose an option Use Information from previous response: order effects Social Desirability Bias Memory (Forgetting) Use information available in the survey context to generate response

58. Improving Reliability: Increase Respondent Motivation Purpose of the Research, its legitimacy Incentives to respond accurately Monetary: $1; Affective: guilt ; Informative: Use research will be put to; Conative: sweepstakes etc. Disentangle collecting data from selling Personal questions at end Include �Other ____� for own perspective

59. Improving Reliability: Reduce Respondent Effort Short questionnaire Provide time it takes to respond. Order of questions should aid memory Provide mnemonic cues: decomposition Most important measure should come first Use closed-scaled responses when possible

60. Improving Reliability: Multiple Measures Ask a question in different ways and assess correlation of responses �Attitude�: �bad-good� �poor-excellent� �Intention�: try, buy, recommend, repurchase Use Multiple scales: open- and closed-ended. Max Price willing to pay/ price evaluation/ quantity @ different prices. Counterbalance order of key measures: General-Specific versus Specific-General Measure respondent motivation and interest

61. Improving Reliability: Use Appropriate Method Does the respondents know the response? Is it �accessible� to them? Can they �explicate� it? Can they �recall� it or merely �recognize� it? Is the respondent willing to share the response? Counter-biasing methods; Anonymity and confidentiality If not, supplement descriptive data with Observation studies Experimental studies

62. Questionnaire Design: Main Take-aways Questionnaire Design is like Advertising The respondent is not interested in answering your questions. You need to make it worth their while, and EASY for them to give you their opinions, behaviors. Questionnaire Design is a Science Use information about memory-models and attitude models to generate your questions. Questionnaire Design is an Art There is no substitute for experience, so Carry out a pretest, revise and then do final draft

63. Topics Questionnaire Design Paper Prototypes

64. Why Do We Prototype ? Get feedback on our design faster Saves money Experiment with alternative designs Fix problems before producing the final product Keep the design centered on the user

65. Fidelity in Prototyping Fidelity refers to the level of detail High fidelity Prototypes look like the final product Low fidelity Artists renditions with many details missing

66. Low-Fi Storyboards Where do storyboards come from? Film & animation Give you a �script� of important events Leave out the details Concentrate on the important interactions In design, the storyboard is non-linear to support user action choices

67. Why Use Low-fi Prototypes? Traditional methods take too long. Sketches -> prototype -> evaluate -> iterate Can simulate the prototype Sketches ->Evaluate -> Iterate Sketches act as prototypes Designer �plays the control unit� Other team members observe and record Sounds silly at first, but is surprisingly effective Kindergarten implementation skills Widely used in industry

68. Hi-Fi Disadvantages Distort perceptions of tester Formal representation indicates �finished� nature People comment on color, fonts, and alignment Discourages major changes Testers don�t want to change �a finished� design Designers don�t want to loss effort put into creating hi-fi design Time is lost on details Discussion tends to be swallowed up on details, not the big-picture issues that matter most.

69. Constructing the Model Draw a window frame on large paper Put different screen regions on cards Anything that moves, changes, appears or disappears Use greek-ing for text if needed Squiggles stand for text not written yet Use photocopier to make many versions

70. Summary Questionnaire Design Low-fi/Paper Prototyping

71. Next Time Design Patterns Design Pattern Languages, Jan Borchers, Chap 2 of �A Pattern Approach to Interaction Design� The Contextual Inquiry and Task Analysis Assignment is out Due in two weeks Get an early start!