slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Chapter 4: Design of Experiments PowerPoint Presentation
Download Presentation
Chapter 4: Design of Experiments

Loading in 2 Seconds...

play fullscreen
1 / 97

Chapter 4: Design of Experiments - PowerPoint PPT Presentation


  • 171 Views
  • Uploaded on

6. Chapter 4: Design of Experiments. 6. Chapter 4: Design of Experiments. Objectives. Explain the role of experiments in answering business questions. You Need to Know. Work is full of questions that you need answers to. Some have answers that only require a lookup:

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Chapter 4: Design of Experiments


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

6

Chapter 4: Design of Experiments

slide2

6

Chapter 4: Design of Experiments

objectives
Objectives
  • Explain the role of experiments in answering business questions.
you need to know
You Need to Know
  • Work is full of questions that you need answers to.
  • Some have answers that only require a lookup:
    • What is the policy regarding the use of demographic variables in predictive models?
    • When did you last send a marketing e-mail to segment 17?
  • Some do not have readily available answers :
    • Does it really matter whether you use first-class postage when sending direct mailings for a cruise line?
    • How should you advertise if you want to maximize sales/expenditure ratio for football tickets?
statistical models can answer questions
Statistical Models Can Answer Questions
  • The models that you learn to use in this course can answer many of the questions that you have.
    • Do you have the data to perform an analysis and answer the question?
    • Did you account for the kinds of variables that are in your control as well as the kind of variables over which you have no control?
questions often mean comparing things
Questions Often Mean Comparing Things
  • Does your question imply that a comparison is needed?
    • First-class versus bulk-rate postage
    • Primetime versus late-night advertising
  • Did you conduct an experiment?
consider this
Consider This…

What is the question that you want to answer?

What is the population that you want the answer to pertain to?

What kinds of things do you want to compare that you can control?

How is the outcome measured (Yobs)?

What else impacts Yobs that you cannot control?

consider this1
Consider This…
  • What is the question that you want to answer?

1. Does postage make a difference in the response rate?

2. Is it worth the extra expense to advertise tickets for a football game in primetime?

consider this2
Consider This…

What is the question that you want to answer?

What is the population that you want the answer to pertain to?

The “luxury traveler” segment

Football fans

consider this3
Consider This…

What is the question that you want to answer?

What is the population that you want the answer to pertain to?

What kinds of things do you want to compare that you can control?

The class of postage on the offer envelope

Whether the tickets are advertised during primetime (expensive) or late night (inexpensive)

consider this4
Consider This…

What is the question that you want to answer?

What is the population that you want the answer to pertain to?

What kinds of things do you want to compare that you can control?

How is the outcome measured (Yobs)?

The number of responses from each postage group

Ticket sales in the week following each type of advertisement

consider this5
Consider This…

What is the question that you want to answer?

What is the population that you want the answer to pertain to?

What kinds of things do you want to compare that you can control?

How is the outcome measured (Yobs)?

What else impacts Yobs that you cannot control?

Gender, vacation already taken that year, children

Team’s season performance (wins, losses), disposable income of viewing markets, broadcasting lineup

who cares about things you cannot control
Who Cares about Things You Cannot Control?

You do!

Only accounting for the things in the experiment that you can control:

who cares about things you cannot control1
Who Cares about Things You Cannot Control?

You do!

Accounting for the things in the experiment that you can control plus one thing that you cannot control:

who cares about things you cannot control2
Who Cares about Things You Cannot Control?

You do!

Accounting for the things in the experiment that you can control plus two things that you cannot control:

consider this6
Consider This…

What is the question that you want to answer?

What is the population that you want the answer to pertain to?

What kinds of things do you want to compare that you can control?

How is the outcome measured (Yobs)?

What else impacts Yobs that you cannot control?

Work smarter: design an experiment!

idea exchange
Idea Exchange
  • Have you ever conducted an experiment? If so, what was the business or scientific objective?
  • Web-based experiments are popular because they are relatively inexpensive to implement and they can be modified in real time. Can you describe any Web experiments you have seen?
  • What kinds of factors might influenceclick-through behavior on, for example, an ad for insurance? For retailclothing? Other types of products and services?
slide18

6

Chapter 4: Design of Experiments

objectives1
Objectives
  • Define experimental design concepts and terminology.
  • Relate experimental design concepts and terminology to business marketing concepts and terminology.
basic terms in design of experiments doe
Basic Terms in Design of Experiments (DOE)

Response

Balance

Factor

Replication

Factor Level

Effect

Orthogonality

Power

Treatment

Experimental Unit

basic terms in doe response
Basic Terms in DOE: Response

A response is the dependent variable of interest in the analyses. It is sometimes called the target or dependent variable.

Examples include the following:

  • Response rate to direct mail solicitations
  • Default (“Bad”) rate among credit customers
  • Balance transfer amount
  • Fraud
  • Number of items purchased from a catalog
  • Spend, six months after acquisition
basic terms in doe factor
Basic Terms in DOE: Factor
  • A factor is an independent variable that is a potential source of variation in the response metric.
  • Examples include the following:
    • Teaser or introductory APR
    • Color of envelope
    • Balance transfer fee
    • Presence or absence of a sticker on a catalog
    • First-class versus third-class mail
    • Others?
basic terms in doe factor level
Basic Terms in DOE: Factor Level
  • A factor level is a particular value, or setting, of a factor.
  • Examples include the following:
    • 1.99% introductory APR
    • White envelope
    • 2% balance transfer fee
    • Airline mile reward offer
    • Third-class mail
    • Others?
basic terms in doe effect
Basic Terms in DOE: Effect

An effect captures and measures the relationship between changes in factor levels and changes in the response metric.

examples of an effect
Examples of an Effect
  • A offer with a sticker on it garners $10 more, in purchases, than a offer without.
examples of an effect1
Examples of an Effect
  • A offer with a sticker on it garners $10 more, in purchases, than a offer without.
  • The white envelope has a 22% higher response rate than the grey envelope.
slide27

A 1% increase in Introductory APR yields a 20% decrease in response rate.

  • A offer with a sticker on it garners $10 more, in purchases, than a offer without.
  • The white envelope has a 22% higher response rate than the grey envelope.
basic terms in doe treatment
Basic Terms in DOE: Treatment
  • A treatment is a combination of all of the factors, each at one level. In a typical marketing context, a treatment constitutes a unique offer.
  • Examples include the following:
    • 1.99% Intro Rate, in a White Envelope, no BT Fee
    • 0% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 1.99% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 0% Intro Rate, in a White Envelope, no BT Fee
  • There are eight possible treatments when you have three factors, each at two levels.
basic terms in doe treatment1
Basic Terms in DOE: Treatment
  • A treatment is a combination of all of the factors, each at one level. In a typical marketing context, a treatment constitutes a unique offer.
  • Examples include the following:
    • 1.99% Intro Rate, in a White Envelope, no BT Fee
    • 0% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 1.99% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 0% Intro Rate, in a White Envelope, no BT Fee
  • There are eight possible treatments when you have three factors, each at two levels.
basic terms in doe treatment2
Basic Terms in DOE: Treatment
  • A treatment is a combination of all of the factors, each at one level. In a typical marketing context, a treatment constitutes a unique offer.
  • Examples include the following:
    • 1.99% Intro Rate, in a White Envelope, no BT Fee
    • 0% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 1.99% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 0% Intro Rate, in a White Envelope, no BT Fee
  • There are eight possible treatments when you have three factors, each at two levels.
basic terms in doe treatment3
Basic Terms in DOE: Treatment
  • A treatment is a combination of all of the factors, each at one level. In a typical marketing context, a treatment constitutes a unique offer.
  • Examples include the following:
    • 1.99% Intro Rate, in a White Envelope, no BT Fee
    • 0% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 1.99% Intro Rate, in a Grey Envelope, 2% BT Fee
    • 0% Intro Rate, in a White Envelope, no BT Fee
  • There are eight possible treatments when you have three factors, each at two levels.
other terms in doe
Other Terms in DOE
  • An experimental unit is the smallest unit to which a treatmentcan be applied.
  • Replication occurs when more than one experimental unitreceives the same treatment.
  • Power is the probability that you will detect an effect, if one exists.
slide33

6

Chapter 4: Design of Experiments

objectives2
Objectives
  • Define multifactor experiments.
  • State the advantages of multifactor experiments versus a sequence of one-factor-at-a-time (OFAT).
  • Explain how experimental units should be allocated to the treatments.
  • Define the term interaction.
  • Analyze a simple multifactor experiment and identify interactions.
two factors each at two levels
Two Factors, Each at Two Levels
  • Example: Credit card solicitation with an introductory, or teaser, rate
    • The introductory (Intro) rate is High or Low.
    • The go-to (Goto) rate is High or Low.
one factor at a time

0% Intro 2.99%

4.99%

Goto

7.99%

One Factor at a Time

Goto Test

Intro = ??

Intro Test

Goto = ??

...

one factor at a time1

4.99%

Goto

7.99%

One Factor at a Time

Intro Test

Hold Goto constant at 4.99%

0% Intro 2.99%

Goto Test

Hold Intro constant at 0%

one factor at a time2

4.99%

Goto

7.99%

One Factor at a Time

"Intro Test"

"Control"

0% Intro 2.99%

"Goto Test"

typical volumes

4.99%

Goto

7.99%

Typical Volumes

50,000 experimental units

50,000 experimental units

0% Intro 2.99%

50,000 experimental units

efficiency
Efficiency
  • VP of Marketing
  • Either a large numerator
  • or a small denominator
  • or both!
  • Experiment Designer
  • Can you quantify these terms?
  • Number of items tested
  • Margin of error
  • Financial costs
  • Total sample size

...

efficiency1
Efficiency
  • VP of Marketing
  • Either a large numerator
  • or a small denominator
  • or both!
  • Experiment Designer
  • Can you quantify these terms?
  • Number of items tested
  • Total sample size
efficiency2
Efficiency
  • VP of Marketing
  • Either a large numerator
  • or a small denominator
  • or both!

Experiment Designer

Can you quantify these terms?

  • Two terms: Intro effect and Goto effect
  • 150,000 observations

...

efficiency3
Efficiency?!?

VP of Marketing

Either a large numerator

or a small denominator

or both!

Experiment Designer

Can you quantify these terms?

  • Two terms: Intro effect and Goto effect
  • 150,000 observations

...

efficiency4

4.99%

Goto

7.99%

Efficiency?!?

0% Intro 2.99%

efficiency5

0% Intro 2.99%

4.99%

Goto

7.99%

Efficiency?!?

This test uses only two-thirds of the data.

This test uses only two-thirds of the data.

one factor at a time3

0% Intro 2.99%

0% Intro 2.99%

4.99%

4.99%

4.99%

Goto

7.99%

7.99%

7.99%

0% Intro 2.99%

0% Intro 2.99%

One Factor at a Time
  • There are many different ways to arrange the “same” test.
  • They all assume no interaction between Intro and Goto.
  • None of these eliminates the potential for bias in the estimates.

Goto

4.99%

Goto

Goto

7.99%

pick a treatment set

0% Intro 2.99%

0% Intro 2.99%

Goto

4.99%

4.99%

4.99%

Goto

7.99%

7.99%

7.99%

4.99%

0% Intro 2.99%

0% Intro 2.99%

Goto

7.99%

Pick a Treatment Set

Goto

detecting interactions between factors
Detecting Interactions between Factors

Low Goto

ResponseRate

High Goto

Low

High

Intro Rate

factorial arrangement of the treatments

0% Intro 2.99%

Goto

4.99%

7.99%

4.99%

0% Intro 2.99%

Goto

7.99%

Factorial Arrangement of the Treatments
    • Permits the testing and estimation of an Intro x Goto interaction term.
    • Increases the precision of estimates for the same test volumes.
    • Can use every individual in every test.
  • Combinations of factor levels provide replication for individual factors.
efficiency reuse observations

0% Intro 2.99%

Goto

4.99%

7.99%

4.99%

0% Intro 2.99%

Goto

7.99%

Efficiency! Reuse Observations
  • The Intro test uses every observation.
efficiency reuse observations1

0% Intro 2.99%

Goto

4.99%

7.99%

4.99%

0% Intro 2.99%

Goto

7.99%

Efficiency! Reuse Observations
  • The Goto test uses every observation.
efficiency additional tests

0% Intro 2.99%

Goto

4.99%

7.99%

4.99%

0% Intro 2.99%

Goto

7.99%

Efficiency! Additional Tests
  • Having four treatment means yields up to four model df.
  • This treatment structure enables the estimation of the Intro x Goto interaction term.
efficiency same or smaller sample size

0% Intro 2.99%

Goto

4.99%

7.99%

4.99%

0% Intro 2.99%

Goto

7.99%

Efficiency! Same or Smaller Sample Size

...

  • Instead of the OFAT approach, with 50,000 experimental units in each treatment (and 1/3 of that data being ignored at each stage of the analysis), this test would require 50,000 observations in each marginal total to have the same power.
efficiency same or smaller sample size1
Efficiency! Same or Smaller Sample Size

25,000

25,000

25,000

25,000

50,000 Low

50,000 High

...

  • Instead of the OFAT approach, with 50,000 experimental units in each treatment (and 1/3 of that data being ignored at each stage of the analysis), this test would require 50,000 observations in each marginal total to have the same power.
efficiency same or smaller sample size2
Efficiency! Same or Smaller Sample Size

25,000

25,000

50,000 Low

50,000 High

25,000

25,000

...

  • Instead of the OFAT approach, with 50,000 experimental units in each treatment (and 1/3 of that data being ignored at each stage of the analysis), this test would require 50,000 observations in each marginal total to have the same power.
efficiency same or smaller sample size3
Efficiency! Same or Smaller Sample Size

25,000

25,000

25,000

25,000

100,000 Total

  • Instead of the OFAT approach, with 50,000 experimental units in each treatment (and 1/3 of that data being ignored at each stage of the analysis), this test would require 50,000 observations in each marginal total to have the same power.
efficiency6
Efficiency?

50,000

50,000

...

  • Balance of the marginal totals might not be all that is required.
efficiency7
Efficiency?

49,999

1

1

49,999

...

  • Balance of the marginal totals might not be all that is required.
efficiency8
Efficiency?

49,500

500

500

49,500

  • Balance of the marginal totals might not be all that is required.
efficiency is still a balancing act
Efficiency Is Still a Balancing Act

25,000

25,000

25,000

25,000

100,000 Total

  • Balancing the sample size over all of the treatments seems like a reasonable goal.
randomization
Randomization
  • After the treatment structure is defined, the next step is to randomly assign treatments to experimental units. A typical approach to randomization of 100,000 customers to four treatments includes the following steps:
    • Define the population of interest.
    • Select a simple random sample from the population equal to the total samples size – for example,100,000.
    • Randomly partition the sample into four equal groups – for example, 25,000.
    • Assign each group to one of the four treatments.
analyzing a 2 by 2 factorial experiment with interaction
Analyzing a 2-by-2 Factorial Experiment with Interaction

Credit Card Case Study

Task: Use SAS Enterprise Guide to graph, analyze, and interpret the results of the two-factor experiment testing two different levels of intro rate and goto rate.

analyzing a 2 by 2 factorial experiment with no interaction
Analyzing a 2-by-2 Factorial Experiment with No Interaction

Credit Card Case Study

Task: Use SAS Enterprise Guide to graph, analyze, and interpret the results of the two-factor experiment testing two different levels of intro rate and goto rate when no interaction is present.

idea exchange1
Idea Exchange
  • Consider the previous experiment.
    • What attributes of the customer might affect an individual’s likelihood to respond to an offer?
    • How could you use your knowledge of these attributes to improve the study’s design and treatment structure?
    • How could you use your knowledge of the attributes to improve the analysis of the experimental data?
exercise
Exercise

This exercise reinforces the concepts discussed previously.

slide66

6

Chapter 4: Design of Experiments

objectives3
Objectives
  • Explain the concept of orthogonality and why it is important.
  • Explain the concept of blocking and why it is useful.
  • Analyze and interpret a multifactor experiment with blocks.
orthogonality
Orthogonality
  • Another ideal property of an experimental design is orthogonality among the elements of interest. There are at least three ways to think about the importance of this property:
    • Algebraic interpretation – Matrices behave well.
    • Geometric interpretation – Pictures look nice.
    • Statistical interpretation – Estimates have low variance.
factorial arrangement versus ofat
Factorial Arrangement versus OFAT

Cons

- Can be more complicated to set up

  • Can be more complicated to sell to a non-technical audience

Factorial Treatment Structure

Pros

+ Reusesobservations (morepower for fewer exp units)

+ Tests for interactions

+ Guarantees balanced and orthogonal treatment plans

+ Is an efficient way to test many factors

factorial arrangement versus ofat1
Factorial Arrangement versus OFAT

One-Factor-at-a-Time Tests

Cons

+/- Permit simple analysis that could be done with a pencil and paper!

- Do not allow a test for interactions

- Represent an inefficient use of experimental units

Pros

+ Are easy to set up – A/B and Champion/Challenger tests are typical in many industries

+ Might yield lower per-unit printing costs

+ Have clear “control” offer, clear test offers

+ Do not require users to learn new words such as“balance” and “orthogonality”!

blocking
Blocking

It is typical to use the same statistic to test

(H0:pmen= pwomen) as

(H0:pred envelope= pblue envelope).

Are these factors equivalent from the perspective of experimental design?

blocking1
Blocking
  • You cannot controlfeatures of your experimental units:
    • Risk profile
    • Responsiveness
    • Geography
    • Age
    • Gender

Restrictions here are typically features of the population of interest, and are often treated as blocks.

  • You can controlfeatures of the offer you make:
    • Creative
    • Color
    • Pricing
    • Duration of offer
  • Any restrictions are typically self-imposed.
  • These are usually factors in the test, not blocks.
blocking2
Blocking

Blocks are groups of experimental units that are homogeneous in some way. Typically, they represent nuisance variability.

Blocks might or might not be randomly selected.

Because units exist in blocks, rather than being assigned to them, blocks reflect a restriction on the randomization in an experiment.

analyzing an experiment with blocks
Analyzing an Experiment with Blocks

Credit Card Case Study

Task: Incorporate a continuous measure such as risk score into a block/factor in an experiment.

idea exchange2
Idea Exchange
  • Consider the kinds of variables that you have no control over. These variables might be important with some types of product offers but not others. What types of product offers might have different response rates based on the following characteristics?
    • Risk profile
    • Geographic regions such as north, south, east, and west
    • Age
    • Gender
    • Urban, suburban, rural
  • Can you think of others?
statistically well formulated model
Statistically Well-Formulated Model

A*B

A

B

Intercept

A well-formulated model maintains the hierarchy of the terms in the model as model reduction is performed. Terms are removed one-at-a-time and the model is refit beforeremoving any more terms.

statistically well formulated model1
Statistically Well-Formulated Model

A*B*C

A*B

A*C

B*C

A

B

C

Intercept

A well-formulated model maintains the hierarchy of the terms in the model as model reduction is performed. Terms are removed one-at-a-time and the model is refit beforeremoving any more terms.

exercise1
Exercise

This exercise reinforces the concepts discussed previously.

slide83

6

Chapter 4: Design of Experiments

objectives4
Objectives
  • Name several continuous response variables you might encounter in business experiments.
  • Describe issues related to analyzing business experiments with continuous responses.
the response variable
The Response Variable
  • In many business applications, the key target variables of interest are binary, and can be expressed as a proportion:
    • Did the customer purchase a product? (What proportion of customers purchased?)
    • Did the product fail? (What proportion of products failed?)
    • Did the customer churn? (What proportion of customers churned?)
    • Was a purchase fraudulent? (What proportion of purchases are fraudulent?)
    • Was there a claim on the policy? (What proportion of policies have claims?)
the response variable1
The Response Variable
  • It is also common to find continuous responses in business models:
    • Revenue per store
    • Number of new customers following an advertising campaign
    • Customer value per mailing
    • Time until churn
    • Wait time on hold in a call center
    • Expected lifetime for a manufactured product
    • Average profit per SKU
where traditional statistics meet the road
Where Traditional Statistics Meet the Road
  • Ordinary least squares (OLS) regression and ANOVA models (linear models) are designed to handle continuous responses.
  • However, not all continuous responses are suitable for OLS models.
the distribution
The Distribution

Revenue per store

Customer value per mailing

Wait time on hold in a call center

the distribution1
The Distribution

Revenue per store

Customer value per mailing

Wait time on hold in a call center

the distribution2
The Distribution

Revenue per store

Customer value per mailing

Wait time on hold in a call center

experimental design and response type
Experimental Design and Response Type

Design and analysis go hand in hand.

Design the experiment so that the analysis will be easy.

Fortunately, the design of the experiment is not contingent on the type of response variable that the data generates.

The same experimental design can be used for evaluating response rate, customer dollar value, lift in revenue, and many other features, regardless of whether they are continuous or categorical.

how do you analyze these response variables
How Do You Analyze These Response Variables?
  • There are many statistical techniques available for modeling continuous responses that are not suited for either logistic regression or OLS techniques.
  • Advances in computing power and technology make such techniques available for business applications through statistical software.
  • These techniques require in-depth understanding of advanced and specialized statistical concepts, and should be used under the direction of a skilled statistician.
idea exchange3
Idea Exchange

How could you incorporate what you know about the cost and profit resulting from different settings (for example, cost of postage or higher profit from higher APR) to help you design an experiment?

slide94

6

Chapter 4: Design of Experiments

recommended reading
Recommended Reading

Ariely, Dan. “Why Businesses Don’t Experiment.” Harvard Business Review. April 2010. http://hbr.org/2010/04/column-why-businesses-dont-experiment/ar/1

recommended reading1
Recommended Reading

Davenport, Thomas. “How to Design Smart Business Experiments.” Harvard Business Review. February 2009.

http://hbr.org/2009/02/how-to-design-smart-business-experiments/ar/1

recommended reading2
Recommended Reading
  • May, Thornton. 2010. The New Know: Innovation Powered by Analytics. New York: Wiley.
    • Chapters 2 and 3