1 / 36

Experimental Design: Part II

Experimental Design: Part II. MAR 6648: Marketing Research February 8, 2010. Overview. Let’s review experiments! What can experiments do that other techniques can’t? How do experiments get implemented in marketing?. An experiment. An experiment?.

paniz
Download Presentation

Experimental Design: Part II

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Experimental Design: Part II MAR 6648: Marketing Research February 8, 2010

  2. Overview • Let’s review experiments! • What can experiments do that other techniques can’t? • How do experiments get implemented in marketing?

  3. An experiment

  4. An experiment? • The owner of two McDonalds franchises here in Gainesville wants to see if transactions run more quickly if he uses both drive-thru windows or only one. He picks one restaurant to use both windows at all times for a month, and the other he has closed at all times for a month. He finds that the drive-thru that uses both windows has notably faster service times.

  5. An experiment?

  6. An experiment?

  7. A Case Study in Causal Investigation • Broken Windows Theory (Kelling & Wilson, 1982) • “If a window in a building is broken and is left unrepaired, all the rest of the windows will soon be broken” • Implication: People see the building as the sort of place where it is OK to break windows • Implication #2: If you fix the window, people see this as a place where people behave nicely • Broad sweeping implication: If you eliminate petty crimes, people will stop perpetrating major crimes

  8. A Case Study in Causal Investigation • Primary concern: The theory is only interesting if it is causal. Consider the alternatives: • 1. Reverse causation: When you reduce major crimes, people are also less likely to commit minor crimes (e.g., if I stop people from stealing cars, I also stop them from breaking windows) • 2. Third variable: If I remove all the criminals from circulation, there will be fewer broken windows AND fewer stolen cars • Now, given the methodologies we have thus far, how would you try to evaluate the theory?

  9. A Case Study in Causal Investigation • Possibilities: • Qualitative Data • Survey Design • Observational Research • Archival Research/Data Mining

  10. Archival Analysis: • Crime in New York City Primary Conclusion: “… an increase in the size of the police force generates a decrease in robberies and burglaries.” Corman and Mocan, 2000

  11. A Case Study in Causal Investigation • Possibilities: • Qualitative Data • Survey Design • Observational Research • Archival Research/Data Mining • Experimental Design • Experimentation is the conscious manipulation of one or more variables by the experimenter in such a way that its effect on one or more variables can be measured. • The variable being manipulated is called the independent variable (a.k.a. cause). • The variable being measured is called the dependent variable (a.k.a. effect). • Elimination of other possible causal factors: i.e., the research design should rule out the other factors (exogenous variables) as potentially causal ones. • This is typically done through random assignment to condition

  12. Experimental Design: Example 1 • Independent Variable • “Policing Disorder” vs. Control • Dependent Variable • Service calls for five serious crimes

  13. Control Policing Disorder Braga and Bond, 2008

  14. Results But what limitations do we see in this?

  15. Experimental Design Experimenters attached a paper flyer to each bicycle and recorded whether or not people dropped the flyer or took it with them. Control Disorderly Setting Keizer, Lindenberg, & Steg, 2008

  16. Results

  17. Further Experiments But what are the shortcomings of this design?

  18. Key Points • Experimentation is necessary to infer causality • A poorly designed experiment will not allow you to infer causality • A good experiment should achieve: • High internal validity through appropriate choice of experimental design. • High external validity by keeping the experimental setting as close to the real marketing environment as possible. • There is a trade-off here… • A good control group is often a key requirement of a good experimental design.

  19. How should experimentation be used in marketing? Focus is on detecting causal relationships between variables ? New Customer Service Program Customer Satisfaction

  20. Causal Research in Marketing • Many examples of the need for causal effects in Marketing • 4P’s alone... • How do we actually identify causal effects? Key tool: Experiments

  21. How to Run Experiments? • How do we actually run experiments? • In the sciences there is a long history of lab experiments • In a lab it is relatively easy to control external conditions that might affect the validity of the experiment • What is the situation in Marketing?

  22. Experiments in Marketing • Usually take the form of a comparison between a test and (at least one) control group • Experiments are frequently run as field experiments

  23. Example: Price Experiment • A catalog retailer selling women’s apparel • Conducted price experiment to estimate demand curves • One control group and 4 test groups • Each group consisted of 15,000 randomly sampled customers

  24. Version 1 Version 2 Version 3 Version 4 Version 5

  25. The Concept of Validity • Internal Validity: • Refers to the ability of the experiment to unambiguously show a cause and effect relationship, i.e., to what extent can we attribute the effect that was observed to the experimental variable and not other factors? • External Validity: • Refers to the extent to which the results of the experiment can be generalized from the experimental environment to the environment of the decision maker; i.e., the real world • There is a trade-off between internal and external validity, from a managerial perspective.

  26. Test Markets • Many uses: • Controlled introduction of a new product • Change of pricing strategy • Change of product design • Choose representative markets • Often as long as one year • Expensive – but highly informative • Three Types: • Simulated Test Markets • Controlled Test Marketing • Sell-In test Marketing

  27. Simulated Test Marketing (STM) • (a.k.a Laboratory Test Markets) • Simulates an actual test market to estimate • initial purchase rates • ultimate repeat purchase rates • Advantages: Compared to true test markets… • Fast (3 months) • Relatively cheap ($250,000) • Flexible • Impressive accuracy rates • Limitations • assumes preference data and purchase/repurchase decisions are valid predictors of what would actually happen in the market place • convenience sample • attrition • Despite this, laboratory test markets are one of the biggest success stories in market research

  28. Example: Typical STM Procedure Step 1: Recruit Qualified Shoppers Step 2: Background Questions Familiarity, Preferences, Usage • Step 4: Simulated Shopping • Respondents given money • Invited into mock/real store • Where they may buy any item Step 3: Screening of Ads Ad for Target Product + Others • Step 6: Reinterview • Contacted after few weeks • Product attitude, usage, satisfaction • Repurchase (REPURCHASE RATE) • Intended & Actual • Step 5: Debriefing • Choices recorded (TRIAL RATE) • Reasons for (non) purchase • Non buyers given free sample

  29. Controlled Test Marketing • Cities for distribution is prearranged • Purchases of a panel of consumers are monitored through scanner data • Example • IRI BehaviorScan • 3,000 households in 7 cities • ID card presented to supermarket • In-store conditions - price, promotion, displays - controlled and monitored • Device on TV allows channel selection to be monitored and ads to be substituted

  30. Sell-In Test Marketing • Cities where product is sold just as it would be in a national launch • Must gain distribution space • Issues • selecting the test cities • implementing and controlling the test • timing • evaluative measures • costs

  31. Six U.S. Common Test Markets

  32. Downsides to Test Markets?

  33. Other Downsides?

  34. Key Points • Experimentation is very useful in marketing to determine true causal effects of marketing decisions • Test markets are the primary example, although smaller scale experiments are common • Test markets represent a trade off between internal and external validity • They have other downsides, too

  35. Summary • Experiments are awesome, right? • They can demonstrate causality, which makes them useful tools for marketers • When you design an experiment, keep in mind the balance you are looking for with regard to internal and external validity • Test markets give you a lot of external validity, which can be good and bad

More Related