Class 25: Thurs., Dec. 9th

1. Class 25: Thurs., Dec. 9th • Today (Final class): Design of Experiments, summary of course. • Schedule: • Mon., Dec. 13th (5 pm) – Preliminary results from final project due. I will make comments on your preliminary results that you can pick up Tuesday afternoon or after. • Tues., Dec. 14th (5 pm) – Homework 9 due • Tues., Dec. 21st (Noon) – Final project due.

2. Two way Analysis of Variance: Steps in Analysis • Check assumptions (constant variance, normality, independence). If constant variance is violated, try transformations. • Fit multiple regression model with an interaction between the two factors. Use effect test to test whether there is evidence of an interaction. 3A. If there is evidence of an interaction, use LS Means plot for interaction variable to display means for different groups. 3B. If there is no evidence of an interaction, refit multiple regression model without the interaction term. Use LS Means plot for each factor’s variable to display how means depend on the factor levels and use effect test to test whether there is evidence that the mean changes as the levels of each factor change (tests for main effects). 4. Create a single factor that combines the two factors and use Tukey’s HSD procedure to make pairwise comparisons of groups, taking into account that multiple comparisons are being done.

3. Statistical Design of Experiments • Scientific Method: • Experimental Design is one of the major contributions of statistics to solving scientific problems.

4. Salk Vaccine Field Trial • Polio was one of the most frightening diseases in the 1930-1950’s. • It struck hardest at young children, causing deaths and leaving many helpless cripples. • Polio appeared in epidemic waves, leading to summer seasons in which communities felt compelled to close swimming pools and restrict public gatherings. • By the 1950s, several vaccines against the disease had been discovered. The one developed by Jonas Salk seemed to be the most promising. In laboratory trials, it had proved safe and had caused production of antibodies against polio. • But it needed to be determined whether vaccine would actually prevent polio in exposed individuals. • Public Health Service and National Foundation for Infantile Paralysis (NFIP) determined that a large scale “field” trial should be undertaken.

5. Proposed Designs for Salk Vaccine Field Trial • Historical Control Approach: Distribute the vaccine as widely as possible, through the schools, to see whether the rate of reported polio was appreciably less than usual during the subsequent season. • Observed Control Approach: Offer vaccination to all children in the second grade of participating schools and follow the polio experience not only in these children but in the first and third grade children. • Randomized, Double Blinded Approach: Choose the control group from the same population as the treatment group – children whose parents consented to vaccination. Assign the treatment randomly. Give a placebo to control group. Do not tell doctors which group children belong to.

6. Problems with the Historical Control Approach • Polio incidence varies widely from season to season, without any attempt to modify it. • If trial had been conducted in 1931 with a placebo vaccine, drop in cases in 1932 would have suggested vaccine was effective.

7. Observed Control Approach • NFIP recognized problem that conditions for historical control group might not be comparable conditions for current group. Wanted to introduce control group similar in characteristics to vaccinated group. • NFIP plan: Offer vaccination to all children in second grade of participating schools and follow polio experience not only in those children but in first and third grade children as well. • Treatment group: vaccinated second graders, Control group: First and third graders.

8. Problems with Observed Control Approach • Of the million children offered vaccination, a half million parents of children refused to have their child vaccinated. Parents who refused to have child vaccinated tended to be less well educated and have lower incomes. This means income and education would be potential lurking variables since control group (all first and third graders) has higher income and education, and income/education are potentially associated with polio.

9. Problems with Observed Control Approach 2. Unblinded diagnosis: In many cases, diagnosis of polio is not clear-cut. Fever and weakness are common symptoms of many illnesses. Under proposed study design, physicians in study areas would have been aware of fact that only second-graders were offered vaccine, and knowledge of whether the child had been vaccinated might have affected the doctor’s diagnosis.

10. Randomized Double Blinded, Controlled Trial • The ideal experiment is a randomized, double blind controlled trial • Control: Compare the active treatment to a control group. Control the experiment so that influences other than treatment operate equally on both control and treatment group, e.g., use contemporaneous rather than historical control group. • Double Blinding: Blind both the subjects and those evaluating the responses (children and doctor here) to which group the subjects belong to. Blinding is achieved by the use of a placebo. • Randomization: Randomly assign the subjects to the treatment and control group.

11. Role of Randomization • Randomization produces groups that are on average comparable in every respect. • Randomization eliminates lurking variables: Remember an omitted variable is only a lurking variable if it associated with both the response and explanatory variable of interest. An omitted variable (e.g., income) cannot be on average associated with whether child receives vaccination if vaccination is randomly assigned (although the particular random assignment could have produced imbalance in income between groups). • An observed difference between the treatment and the control group must be due either to the treatment or to the play of chance in the random assignment of subjects to the groups. • Statistical inference (p-values, confidence intervals) provides a method for describing how confident we can be that an observed difference between the treatment and control group did not arise due to chance.

12. Salk Vaccine Field Trial Results • Rates of polio cases per 100,000 in each group • Randomized Controlled Double-Blind Experiment: • Treatment: 28 • Control: 71 • No Consent: 46 • NFIP Observed Control study: • Grade 2 (vaccine): 25 • Grades 1 and 3 (control): 54 • Grade 2 (no consent): 44 • Comparison of treatment to control shows larger effect in randomized controlled double-blind experiment. The no consent group in observed control group appears healthier than the consent group (Grade 2 – nonconsent rate > Grades 1 and 3 control rate).

13. Design of Experiments: Replication • In order to reduce the role of chance variation in random assignment, we want to replicate each treatment on many units. • How to choose the sample size needed to obtain reliable results that will not likely be affected by chance variation?

14. Errors in Hypothesis Testing When we do one hypothesis test and reject null hypothesis if p-value <0.05, then the probability of making a Type I error when the null hypothesis is true is 0.05. We protect against falsely rejecting a null hypothesis by making probability of Type I error small.

15. Power Calculations • The probability of rejecting the null hypothesis when the alternative hypothesis is true is called the power. • The power depends on the specific alternative hypothesis is true. • In designing an experiment, we want to choose the sample size so that the experiment has high power (say power > 0.8) for an alternative hypothesis that is considered to be of practical importance. • Why is it important for a study to have high power: If an experiment has low power, then even if the alternative hypothesis is true, we will not have a large chance to detect it. We will not have learned anything from the experiment. • In order to do a power calculation, we need to specify what hypothesis test will be done, what alternative hypothesis we want to calculate the power for and typically what the standard deviation of the population is.

16. Choosing Sample Size Example • Does increasing the amount of calcium in our diet reduce blood pressure? A randomized, controlled, double blind experiment is planned with a treatment group that will receive a calcium supplement for 12 weeks and a control group that will receive a placebo. The seated systolic blood pressure will be measured at the beginning and end of the 12-week period and the response variable will be the difference in blood pressure between the end and beginning of the 12-week period. The standard deviation of change in blood pressure over a 12-week period in the population is 7.4. • The researchers would like to be able to detect a difference of 5 in the mean difference of blood pressure between the beginning and end of the 12 week period between the treatment and control groups? Will 30 subjects in the treatment and control groups (60 subjects total) be enough to have high power for the alternative hypothesis of a difference of 5 in the mean difference of blood pressure between the beginning and end of the 12-week period between the treatment and control groups?

17. Design of Experiments in JMP • JMP calculates power for a variety of different experiments by clicking DOE, Sample Size and Power • For a comparisons of two groups via the one-way ANOVA F test (effect test), the power of an experiment to reject the null hypothesis that the two groups have the same means is computed by clicking DOE, Sample Size and Power, Two Sample Means, then putting the expected standard deviation in Error Std Dev box, the differences in the mean of the two groups that one wants to compute the power for in Difference to Detect box and the sample size in Sample Size box. Click Continue, power appears in the power box. • To choose the minimum sample size to give a certain power, leave the Sample Size box blank and put the desired power in the Power box and Click continue. Minimum sample size needed appears in the Sample Size box.

18. Sample size of 60 (30 in each group) has power 0.73; we’d like an experiment to have power at least 0.8 for the alternative hypothesis of interest. Sample size of 71 is the smallest sample size that provides a power of at least 0.8.

19. Ethics of Experiments • Is using a placebo ethical? • Is doing a bad study (e.g., one that has low power) ethical? • Is misleading subjects ethical (e.g., in Milgram’s study) ethical?

20. Summary of Course • Statistical methods learned: Simple regression, multiple regression, one-way and two-way analysis of variance, design of experiments • Important points to remember from course: • Regression is a useful tool for describing the data you have collected and prediction, but must be used carefully when trying to establish causation. Beware of lurking variables. • Estimates of regression coefficients should always be accompanied by confidence intervals. The confidence intervals tells us what the plausible values of the true coefficients are based on the data. • Experiments are the best way to establish causation. Choose sample size so that the experiment has adequate power to detect alternative hypothesis of interest.

21. Directions for Future Study • Stat 210: Sample Survey Design. Offered next spring. Good follow-up to this course for those interested in social science. • Stat 500: Applied Regression and Analysis of Variance. Offered next fall. Natural follow-up to Stat 112, giving a more advanced treatment of the topics in 112. • Stat 501: Introduction to Nonparametric Methods and Log-linear models. Offered this spring. Follow-up to Stat 500. • Stat 430: Probability. Will be offered this spring and next fall. • Stat 431: Statistical Inference. Will be offered this spring and next fall.