Research Methods: An Introduction

1. Research Methods: An Introduction Priya Raghubir Haas School of Business

2. Validity, Reliability, Representativeness • Validity: Asking the right question • Reliability: Receiving the right response • Reducing Response Error • Representativeness: Generalizing results. • Sample selection and sample size • Reducing Non-response Error Berkeley Consulting

3. Reliability • Similar results by comparable measures of the same construct • Two ways to measure reliability • Test-Retest:: stability over time • Split-Half: consistency across ratings within a multiple-item scale • Reliability is easier to determine than Validity • If a measure is valid, then it is reliable • If it is not reliable, it can not be valid • If it is reliable, it may or may not be valid Berkeley Consulting

4. Introducing New Coke: Research Conducted • 2000 interviews in 10 major markets: • Control for brand names; Storyboards and Mock Advertisements: Try? Switch? Be Upset? • Focus Groups: Mixed; loyal drinkers unfavorable. • Blind Taste Tests: n = 30,000-40,000 • New vs. Old Coke: 55 vs. 45; vs. Pepsi: 54-46 • Generalizability: n=191,000 in 13 cities; Blind taste tests of 4 new vs. old / Pepsi. Berkeley Consulting

5. New Coke: Key Research Design Takeaways • Predictive Validity • Did not research impact of the Loss of Coke • Construct Validity: • Buy brand vs. prefer taste? • Reliability: • Discriminate/ Prefer Berkeley Consulting

6. New Coke: Key Research Implementation Takeaways • Sample Size: • Effect Size vs. Significance • Data Analysis: • Ignoring focus group dissent • Conclusions: • 39% did not prefer new taste in branded taste test; 45% in blind taste test! • But then, there is Hindsight Bias … Berkeley Consulting

7. Biases In Answering Survey Questions Basic tenet: Ask a question, and you’ll get a response! Translation: Garbage in, garbage out! Berkeley Consulting

8. Examples of Questions from Consumer Surveys (The Wall Street Journal, Nov. 14, 1991) • When Levi Strauss & Co. asked students which clothes would be most popular this year, 90% said Levi’s 501 jeans. (They were the only jeans on the list.) • A survey for Black Flag said: “A roach disk… poisons a roach slowly. The dying roach returns to the nest and after it dies is eaten by other roaches. In turn, these roaches become poisoned and die. How effective do you think this type of product would be in killing roaches?” Not surprisingly, 79% said effective! Berkeley Consulting

9. Examples…(continued from previous slide) • A Gallup poll sponsored by the disposable-diaper industry asked: “It is estimated that disposable diapers account for less than 2% of the trash in today’s landfills. In contrast, beverage containers, third-class mail and yard waste to account for about 21% of the trash in landfills. Given this, in your opinion, would it be fair to ban disposable diapers?” (84% said no.) Berkeley Consulting

10. Sources of Error in Surveys • Non-response bias due to refusals • fear of consequences of participation, represents an invasion of privacy, anxiety about the subject • Inaccurate responses • inability to respond: ignorance, memory problems, problems formulating an answer • unwilling to respond accurately: concerns about invasion of privacy, time pressure/fatigue, prestige seeking/social responsibility, response style/bias • Error causes by interviewers • providing clues to answers • inadequate questioning or probing Berkeley Consulting

11. Reducing Response Errors • How do respondents answer survey questions? • Guess • Randomly choose an option • Use Information from previous response: order effects • Social Desirability Bias • Memory (Forgetting) Use information available in the survey context to generate response Berkeley Consulting

12. Improving Reliability: Increase Respondent Motivation • Purpose of the Research, its legitimacy • Incentives to respond accurately • Monetary: $1; Affective: guilt ; Informative: Use research will be put to; Conative: sweepstakes etc. • Disentangle collecting data from selling • Personal questions at end • Include “Other ____” for own perspective Berkeley Consulting

13. Improving Reliability: Reduce Respondent Effort • Short questionnaire • Provide time it takes to respond. • Order of questions should aid memory • Provide mnemonic cues: decomposition • Most important measure should come first • Use closed-scaled responses when possible Berkeley Consulting

14. Improving Reliability: Multiple Measures • Ask a question in different ways and assess correlation of responses (Cronbach’s ) • “Attitude”: “bad-good” “poor-excellent” • “Intention”: try, buy, recommend, repurchase • Use Multiple scales: open- and closed-ended. • Max Price willing to pay/ price evaluation/ quantity @ different prices. • Counterbalance order of key measures: • General-Specific versus Specific-General • Measure respondent motivation and interest Berkeley Consulting

15. Improving Reliability: Use Appropriate Method • Does the respondents know the response? • Is it “accessible” to them? • Can they “explicate” it? • Can they “recall” it or merely “recognize” it? • Is the respondent willing to share the response? • Counter-biasing methods; • Anonymity and confidentiality • If not, supplement descriptive data with • Observation studies • Experimental studies Berkeley Consulting

16. The New York Times, 7/8/94 Berkeley Consulting

17. How to ask a sensitive question(Barton, POQ 1958) • The casual approach: "Do you happen to have killed your partner?” • The numbered card approach: "Would you please read off the number on this card which corresponds to what became of your partner? 1 Natural death 2 I killed her/ him 3 Other" Berkeley Consulting

18. How to ask a sensitive question(Barton, POQ 1958; continued) • The everybody approach: (also called counterbiasing method) "As you know, many people have been killing their partners these days. Do you happen to have killed yours?" • The other people approach: "Do you know any people who have murdered their partners? How about yourself?" Berkeley Consulting

19. Sensitive questions(continued) • Personal Information - Build rapport, legitimize • Embarrassing Information (downward bias) • counterbiasing statement: “Recent studies indicate that a man often uses his wife’s hair spray. Do you use your wife’s hair spray?” • projective techniques • randomized response techniques • Prestige, Normative Information (upward bias) Berkeley Consulting

20. Asking Sensitive Questions: Rules of Thumb • Hide in a group of more innocuous questions • State that the behavior or attitude is not unusual • Phrase in terms of others and how they might feel or react • Use multiple-choice response categories • Randomized response Berkeley Consulting

21. Watch Out: Garbage In, Garbage Out • Behavioral Questions: Memory Biases • Omission • Telescoping • Attitude Questions: Confusion • Double-barreled, leading, ambiguous questions • Sensitive Questions: “Lying” • Item non-repsonse • Termination • Distortion Berkeley Consulting

22. Questionnaire Design: Main Take-aways • Questionnaire Design is like Advertising • The respondent is not interested in answering your questions. • You need to make it worth their while, and EASY for them to give you their opinions, behaviors. • Questionnaire Design is a Science • Use information about memory-models and attitude models to generate your questions. • Questionnaire Design is an Art • There is no substitute for experience, so • Carry out a pretest, revise and then do final draft Berkeley Consulting