 Download Download Presentation Outline

# Outline

Download Presentation ## Outline

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
##### Presentation Transcript

1. More Experiment DesignCS 239Experimental Methodologies for System SoftwarePeter ReiherMay 8, 2007

2. Outline • Multiplicative experiment design models • 2kr factorial experiment designs • 2k-p fractional factorial designs • Confounding in fractional factorial designs

3. Multiplicative Models for 22r Experiments • Assumptions of additive models • Example of a multiplicative situation • Handling a multiplicative model • When to choose multiplicative model • Multiplicative example

4. Assumptions of Additive Models • Last time’s analysis used additive model: • yij = q0+ qAxA+ qBxB+ qABxAxB+ eij • Assumes all effects are additive: • Factors • Interactions • Errors • This assumption must be validated!

5. Example of aMultiplicative Situation • Testing processors with different workloads • Most common multiplicative case • Consider 2 processors, 2 workloads • Use 22r design • Response is time to execute wj instructions on processor that takes vi seconds/instruction • wj and vi sound like good factors to test • Without interactions, time is yij = viwj

6. Handlinga Multiplicative Model • Take logarithm of both sides: yij = viwj so log(yij) = log(vi) + log(wj) • Use additive model on logarithms • XA is log(vi), XB is log(wj) • Choose your high and low levels for each • Resulting model is: • log(yij) = q0 +qA XA+ qB XB+ qAB XA XB +eij • But we care about yij, not log(yij)

7. Converting Back to yij • Take antilog of both sides of equation • UA = 10 qA • UB = 10 qB • UAB = 10 qAB

8. Meaning of aMultiplicative Model • Model is • Here, mA = 10qA is ratio of MIPS ratings of processors, mB = 10qB is ratio of workload size • Antilog of q0 is geometric mean of responses:where n = 22r

9. When to Choosea Multiplicative Model? • Physical considerations (see previous slides) • Range of y is large • Making arithmetic mean unreasonable • Calling for log transformation • Plot of residuals shows large values and increasing spread • Quantile-quantile plot doesn’t look like normal distribution

10. Multiplicative Example • Consider additive model of processors A1 and A2 running benchmarks B1 and B2: • Note large range of y values

11. Error Scatterof Additive Model

12. Quantile-Quantile Plotof Additive Model

13. Multiplicative Model • Taking logs of everything, the model is:

14. Error Residuals ofMultiplicative Model

15. Quantile-Quantile Plot forMultiplicative Model

16. Summary ofthe Two Models • Which suggests the time to run a benchmark depends only on the processor speed and benchmark size • Sounds about right

17. General 2kr Factorial Design • Simple extension of 22r • Just k factors, not 2 • See Box 18.1 for summary • Always do visual tests • Remember to consider multiplicative model as alternative

18. Example of 2krFactorial Design • Consider a 233 design • 3 factors • 2 levels for each • 3 replications of each combination • There will be more factor interaction terms, of course

19. Sign Table for a Sample 2krFactorial Design

20. Allocation of Variation for 233 Design • Percent variation explained: • 90% confidence intervals

21. Error Residualsfor 233 Design

22. Quantile-Quantile Plot for All Points for 233 Design

23. Quantile-Quantile Plot for Means 233 • R2 for this one is .94

24. Concerns With These Kinds of Designs • They don’t test all possible levels • Only test two, in fact • Solved by full factorial designs • Which we’ll cover later • They are a lot of work • Especially if there are many factors • Solved by fractional factorial design

25. Fractional Designs • What if there are many factors? • You can’t afford to test all combinations • Well, then, test only some of them • How should you determine which combinations to test? • Losing least information

26. 2k-p FractionalFactorial Designs • Introductory example of a 2k-p design • Preparing the sign table for a 2k-p design • Confounding • Algebra of confounding • Design resolution

27. What Is A 2k-p FractionalFactorial Design? • As before, test only two levels of each factor • But instead of testing all 2k factors, • Only test 2k-p of them • The larger p is, the fewer combinations tested • E.g., for k = 5 and p = 2, reduces tests from 32 to 8

28. Introductory Exampleof a 2k-p Design • Exploring 7 factors in only 8 experiments • k = 7, p = 4 • Full factorial design would take 128 experiments • Won’t we save time! • Would be nice to know what price we paid • We can’t know everything • But we can get some control

29. Sign Table for Example

30. Analysis of 27-4 Design • Column sums are zero: • Sum of 2-column product is zero: • Sum of column squares is 27-4 = 8 • Orthogonality allows easy calculation of effects:

31. Effects and Confidence Intervals for 2k-p Designs • Effects are as in 2k designs: • % variation proportional to squared effects • For standard deviations and confidence intervals: • Use formulas from full factorial designs • Replace 2k with 2k-p

32. Preparing the Sign Table for a 2k-p Design • Start by preparing a sign table for k-p factors • Assign first k-p factors as before • Then assign remaining factors • In the place of some (or all) of the combined effects columns

33. Sign Table for k-p Factors • Same as table for experiment with k-p factors • I.e., 2(k-p) table • 2k-p rows and 2k-p columns • First column is I, contains all 1’s • Next k-p columns get k-p selected factors • Rest are products of factors

34. Assigning Remaining Factors • 2k-p-(k-p)-1 product columns remain • Choose any p columns • Assign remaining p factors to them • Any others stay as-is, measuring interactions

35. An Example • Let’s build a 25-2 table • So there are five factors A, B, C, D, and E • But we only want to run 8 experiments • p = 2 • 5-2=3

37. Now Add the Remaining Factors D and E

38. Our Final Sign Table

39. Running Experiments With This Sign Table • Use it just as before • Run the set of experiments the table indicates • E.g., run A,B,C,D at low level, E at high level • Then A and E at high, B, C, and D at low • And so on

40. Calculating Effects With the Sign Table • Just like before • Multiply experiment results by columns • Add up results • Divide by number of experiments • There are your q values

41. What Have We Paid? • The fourth column shows the combined effects of A and B • The fifth column shows the combined effects of A and C • What about all the other effect combinations?

42. Confounding • The other combined effects were confounded • The confounding problem • An example of confounding • Confounding notation • Choices in fractional factorial design

43. The Confounding Problem • Fundamental to fractional factorial designs • Some effects produce combined influences • Limited experiments mean only some combinations can be calculated • Problem of combined influence is confounding • Inseparable effects called confounded effects

44. An Example of Confounding • Consider this 23-1 table: • Extend it with an AB column:

45. Analyzing theConfounding Example • Effect of C is same as that of AB: qC = (y1-y2-y3+y4)/4 qAB = (y1-y2-y3+y4)/4 • Formula for qC really gives combined effect: qC+qAB = (y1-y2-y3+y4)/4 • No way to separate qC from qAB • Not a problem if qAB is known to be small

46. Let’s Go Back to Our Example • Where are combined effects AD, AE, BC, BD, BE, CD, CE, and DE? • Not to mention ABC, ABD, ABE, ACD, ACE, ADE, BCD, BDE, BCE, CDE, ABCD, ABCE, ACDE, ABDE,BCDE, and ABCDE?

47. Confounding Notation • Previous 23-1 confounding is denoted by equating confounded effects:C = AB • Other effects are also confounded in this design:A = BC, B = AC, I = ABC • Last entry indicates ABC is confounded with overall mean, or q0

48. What Does Confounding Really Mean? • Each effect is a combination of several effects from a full experiment • Impossible to pull out one from the other • Unless you change design and make more runs • Must be aware of what’s getting confounded

49. Getting Concrete on The Meaning of Confounding • Consider our generic 23-1 fractional factorial experiment • What if we’re measuring computer performance? • With three factors: • CPU speed (A) • Memory size (B) • Disk speed (C)

50. Using Our Fractional Design, • We have combined the effect of disk speed with the interaction of CPU speed and memory size (C=AB) • And the effect of CPU speed with the combined effects of disk speed and memory size (A=BC) • Among several others