1 / 68

Outline

More Experiment Design CS 239 Experimental Methodologies for System Software Peter Reiher May 8, 2007. Outline. Multiplicative experiment design models 2 k r factorial experiment designs 2 k-p fractional factorial designs Confounding in fractional factorial designs.

satya
Download Presentation

Outline

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. More Experiment DesignCS 239Experimental Methodologies for System SoftwarePeter ReiherMay 8, 2007

  2. Outline • Multiplicative experiment design models • 2kr factorial experiment designs • 2k-p fractional factorial designs • Confounding in fractional factorial designs

  3. Multiplicative Models for 22r Experiments • Assumptions of additive models • Example of a multiplicative situation • Handling a multiplicative model • When to choose multiplicative model • Multiplicative example

  4. Assumptions of Additive Models • Last time’s analysis used additive model: • yij = q0+ qAxA+ qBxB+ qABxAxB+ eij • Assumes all effects are additive: • Factors • Interactions • Errors • This assumption must be validated!

  5. Example of aMultiplicative Situation • Testing processors with different workloads • Most common multiplicative case • Consider 2 processors, 2 workloads • Use 22r design • Response is time to execute wj instructions on processor that takes vi seconds/instruction • wj and vi sound like good factors to test • Without interactions, time is yij = viwj

  6. Handlinga Multiplicative Model • Take logarithm of both sides: yij = viwj so log(yij) = log(vi) + log(wj) • Use additive model on logarithms • XA is log(vi), XB is log(wj) • Choose your high and low levels for each • Resulting model is: • log(yij) = q0 +qA XA+ qB XB+ qAB XA XB +eij • But we care about yij, not log(yij)

  7. Converting Back to yij • Take antilog of both sides of equation • UA = 10 qA • UB = 10 qB • UAB = 10 qAB

  8. Meaning of aMultiplicative Model • Model is • Here, mA = 10qA is ratio of MIPS ratings of processors, mB = 10qB is ratio of workload size • Antilog of q0 is geometric mean of responses:where n = 22r

  9. When to Choosea Multiplicative Model? • Physical considerations (see previous slides) • Range of y is large • Making arithmetic mean unreasonable • Calling for log transformation • Plot of residuals shows large values and increasing spread • Quantile-quantile plot doesn’t look like normal distribution

  10. Multiplicative Example • Consider additive model of processors A1 and A2 running benchmarks B1 and B2: • Note large range of y values

  11. Error Scatterof Additive Model

  12. Quantile-Quantile Plotof Additive Model

  13. Multiplicative Model • Taking logs of everything, the model is:

  14. Error Residuals ofMultiplicative Model

  15. Quantile-Quantile Plot forMultiplicative Model

  16. Summary ofthe Two Models • Which suggests the time to run a benchmark depends only on the processor speed and benchmark size • Sounds about right

  17. General 2kr Factorial Design • Simple extension of 22r • Just k factors, not 2 • See Box 18.1 for summary • Always do visual tests • Remember to consider multiplicative model as alternative

  18. Example of 2krFactorial Design • Consider a 233 design • 3 factors • 2 levels for each • 3 replications of each combination • There will be more factor interaction terms, of course

  19. Sign Table for a Sample 2krFactorial Design

  20. Allocation of Variation for 233 Design • Percent variation explained: • 90% confidence intervals

  21. Error Residualsfor 233 Design

  22. Quantile-Quantile Plot for All Points for 233 Design

  23. Quantile-Quantile Plot for Means 233 • R2 for this one is .94

  24. Concerns With These Kinds of Designs • They don’t test all possible levels • Only test two, in fact • Solved by full factorial designs • Which we’ll cover later • They are a lot of work • Especially if there are many factors • Solved by fractional factorial design

  25. Fractional Designs • What if there are many factors? • You can’t afford to test all combinations • Well, then, test only some of them • How should you determine which combinations to test? • Losing least information

  26. 2k-p FractionalFactorial Designs • Introductory example of a 2k-p design • Preparing the sign table for a 2k-p design • Confounding • Algebra of confounding • Design resolution

  27. What Is A 2k-p FractionalFactorial Design? • As before, test only two levels of each factor • But instead of testing all 2k factors, • Only test 2k-p of them • The larger p is, the fewer combinations tested • E.g., for k = 5 and p = 2, reduces tests from 32 to 8

  28. Introductory Exampleof a 2k-p Design • Exploring 7 factors in only 8 experiments • k = 7, p = 4 • Full factorial design would take 128 experiments • Won’t we save time! • Would be nice to know what price we paid • We can’t know everything • But we can get some control

  29. Sign Table for Example

  30. Analysis of 27-4 Design • Column sums are zero: • Sum of 2-column product is zero: • Sum of column squares is 27-4 = 8 • Orthogonality allows easy calculation of effects:

  31. Effects and Confidence Intervals for 2k-p Designs • Effects are as in 2k designs: • % variation proportional to squared effects • For standard deviations and confidence intervals: • Use formulas from full factorial designs • Replace 2k with 2k-p

  32. Preparing the Sign Table for a 2k-p Design • Start by preparing a sign table for k-p factors • Assign first k-p factors as before • Then assign remaining factors • In the place of some (or all) of the combined effects columns

  33. Sign Table for k-p Factors • Same as table for experiment with k-p factors • I.e., 2(k-p) table • 2k-p rows and 2k-p columns • First column is I, contains all 1’s • Next k-p columns get k-p selected factors • Rest are products of factors

  34. Assigning Remaining Factors • 2k-p-(k-p)-1 product columns remain • Choose any p columns • Assign remaining p factors to them • Any others stay as-is, measuring interactions

  35. An Example • Let’s build a 25-2 table • So there are five factors A, B, C, D, and E • But we only want to run 8 experiments • p = 2 • 5-2=3

  36. Start With a 23 Table

  37. Now Add the Remaining Factors D and E

  38. Our Final Sign Table

  39. Running Experiments With This Sign Table • Use it just as before • Run the set of experiments the table indicates • E.g., run A,B,C,D at low level, E at high level • Then A and E at high, B, C, and D at low • And so on

  40. Calculating Effects With the Sign Table • Just like before • Multiply experiment results by columns • Add up results • Divide by number of experiments • There are your q values

  41. What Have We Paid? • The fourth column shows the combined effects of A and B • The fifth column shows the combined effects of A and C • What about all the other effect combinations?

  42. Confounding • The other combined effects were confounded • The confounding problem • An example of confounding • Confounding notation • Choices in fractional factorial design

  43. The Confounding Problem • Fundamental to fractional factorial designs • Some effects produce combined influences • Limited experiments mean only some combinations can be calculated • Problem of combined influence is confounding • Inseparable effects called confounded effects

  44. An Example of Confounding • Consider this 23-1 table: • Extend it with an AB column:

  45. Analyzing theConfounding Example • Effect of C is same as that of AB: qC = (y1-y2-y3+y4)/4 qAB = (y1-y2-y3+y4)/4 • Formula for qC really gives combined effect: qC+qAB = (y1-y2-y3+y4)/4 • No way to separate qC from qAB • Not a problem if qAB is known to be small

  46. Let’s Go Back to Our Example • Where are combined effects AD, AE, BC, BD, BE, CD, CE, and DE? • Not to mention ABC, ABD, ABE, ACD, ACE, ADE, BCD, BDE, BCE, CDE, ABCD, ABCE, ACDE, ABDE,BCDE, and ABCDE?

  47. Confounding Notation • Previous 23-1 confounding is denoted by equating confounded effects:C = AB • Other effects are also confounded in this design:A = BC, B = AC, I = ABC • Last entry indicates ABC is confounded with overall mean, or q0

  48. What Does Confounding Really Mean? • Each effect is a combination of several effects from a full experiment • Impossible to pull out one from the other • Unless you change design and make more runs • Must be aware of what’s getting confounded

  49. Getting Concrete on The Meaning of Confounding • Consider our generic 23-1 fractional factorial experiment • What if we’re measuring computer performance? • With three factors: • CPU speed (A) • Memory size (B) • Disk speed (C)

  50. Using Our Fractional Design, • We have combined the effect of disk speed with the interaction of CPU speed and memory size (C=AB) • And the effect of CPU speed with the combined effects of disk speed and memory size (A=BC) • Among several others

More Related