Summer School in Statistics for Astronomers VI June 7-11, 2010 Robustness, Nonparametrics and Some Inconvenient Truths

1. Summer School in Statisticsfor Astronomers VIJune 7-11, 2010Robustness, Nonparametrics and Some Inconvenient Truths Tom Hettmansperger Dept. of Statistics Penn State University

2. Least squares t-tests and F-test Robust methods rank tests Nonparametrics

3. Some ideas we will explore: • Robustness • Nonparametric Bootstrap • Nonparametric Density Estimation • Nonparametric Rank Tests • Tests for (non-)Normality The goal: To make you worry or at least think critically about statistical analyses.

4. Abstract Population Distribution, Model Probability and Expectation Statistical Inference Real World Data

5. Research Hypothesis or Question in English Measurement, Exp. Design, Data Collection Statistical Model, Population Distribution Translate Res. Hyp. or Quest. into a statement in terms of the model parameters Select a relevant statistic Carry out statistical inference Graphical displays Model criticism Sampling Distributions P-values Significance levels Confidence coefficients State Conclusions and Recommendations in English

6. Parameters in a population or model Typical Values: mean, median, mode Spread: variance (standard deviation), interquartile range (IQR) Shape: probability density function (pdf), cumulative distribution function (cdf) Outliers

7. Research Question: How large are the luminosities in NGC 4382? Measure of luminosity (data below) Traditional model: normal distribution of luminosity Translate Res. Q.: What is the mean luminosity of the population? (Here we use the mean to represent the typical value.) The relevant statistic is the sample mean. Statistical Inference: 95% confidence interval for the mean using a normal approximation to the sampling distribution of the mean. orig: 26.905 + .0524 no: 26.917 + .0474 24.000: 26.867 + .1094 NGC 4382 (n = 59) 26.215 26.506 26.542 26.551 26.553 26.607 26.612 26.674 26.687 26.699 26.703 26.727 26.740 26.747 26.765 26.779 26.790 26.800 26.807 ... 27.161 27.169 27.179

10. Variable N N* Mean SE Mean StDev NGC 4382_no 58 0 26.9170.02370.181 NGC 4382_orig 59 0 26.905 0.0262 0.201 NGC 4382_26 59 0 26.901 0.0280 0.215 NGC 4382_25 59 0 26.884 0.0400 0.307 NGC 4382_24 59 0 26.8670.05470.420 Minimum Q1 Median Q3 26.506 26.776 26.974 27.046 26.215 26.765 26.974 27.042 26.000 26.765 26.974 27.042 25.000 26.765 26.974 27.042 24.000 26.765 26.974 27.042

11. First Inconvenient Truth: Outliers can have arbitrarily large impact on the sample mean, sample standard deviation, and sample variance. Second Inconvenient Truth: A single outlier can increase the width of the t-confidence interval and inflate the margin of error for the sample mean. Inference can be adversely affected. It is bad for a small portion of the data to dictate the results of a statistical analysis.

12. Third Very Inconvenient Truth: The construction of a 95% confidence interval for the population variance is very sensitive to the shape of the underlying model distribution. The standard interval computed in most statistical packages assumes the model distribution is normal. If this assumption is wrong, the resulting confidence coefficient can vary significantly. I am not aware of a stable 95% confidence interval for the population variance.

13. The ever hopeful statisticians

14. Robustness: structural and distributional Structural: We would like to have an estimator and a test statistic that are not overly sensitive to small portions of the data. Influence or sensitivity curves: The rate of change in a statistic as an outlier is varied. Breakdown: The smallest fraction of the data that must be altered to carry the statistic beyond any preset bound. We want bounded influence and high breakdown.

15. Distributional robustness: We want a sampling distribution for the test statistic that is not sensitive to changes or misspecifications in the model or population distribution. This type of robustness provides stable p-values for testing and stable confidence coefficients for confidence intervals.

18. Message: The sample mean is not structurally robust; whereas, the median is structurally robust. It takes only one observation to move the sample mean anywhere. It takes roughly 50% of the data to move the median. (Breakdown) Sensitivity Curve: SCmean(x) = x SCmedian(x) = (n+1)x(r) if x < x(r) (n+1)x if x(r) < x < x(r+1) (n+1)x(r+1) if x(r+1)< x when n = 2r

19. Influence Mean Median x Mean has linear, unbounded influence. Median has bounded influence.

20. Some good news: The sampling distribution of the sample mean depends only mildly on the population or model distribution. (A Central Limit Theorem effect) Provided our data come from a model with finite variance, for large sample size has an approximate standard normal distribution (mean 0 and variance 1). This means that the sample mean enjoys distributional robustness, at least approximately. We say that the sample mean is asymptotically nonparametric.

21. More inconvenient truth: the sample variance is neither structurally robust (unbounded sensitivity and breakdown tending to 0), but also lacks distributional robustness. Again, from the Central Limit Theorem: Provided our data come from a model with finite fourth moment, for large sample size has an approximate normal distribution with mean 0 and variance: g is called the kurtosis

22. The kurtosis and is a measure of the tail weight of a model distribution. It is independent of location and scale and has value 3 for any normal model. Assuming 95% confidence:

23. A very inconvenient truth: A test for normality will also mislead you!!

24. Some questions: • If statistical methodology based on sample means • and sample variances is non robust, what can we do? • Are you concerned about the last least squares analysis • you carried out? (t-tests and F-tests) If not, you should be! • What if we want to simply replace the mean by the • median as the typical value? The sample median is • robust, at least structurally. What about the distribution? • The mean and the t-test go together. What test goes • with the median?

25. We know that: How to find SE(median) and estimate it. • Two ways: • Nonparametric Bootstrap (computational) • Estimate the standard deviation of the • approximating normal distribution. (theoretical)

26. Nonparametric Bootstrap; • 1. Draw a sample of size 59 from the original NGC4382 • data. Sample with replacement. • Compute and store the sample median. • Repeat B times. (I generally take B = 4999) • The histogram of the B medians is an estimate of • sampling distribution of the sample median. • Compute the standard deviation of the B medians. • This is the approximate SE of the sample median. Result for NGC4382: SE(median) = .028 (.027 w/o the outlier, .028 w outlier = 24)

27. Theoretical (Mathematical Statistics) Moderately Difficult Let M denote the sample median. Provided the density (pdf) of the model distribution is not 0 at the model median, has an approximate normal distribution with mean 0 and variance 1/[4f2(q)]. where f(x) is the density and q is the model median. In other words, SE(median) = and we must estimate the value of the density at the population median.

28. Nonparametric density estimation: Let f(x) denote a pdf. Based on a sample of size n we wish to estimate f(x0) where x0 is given. Define: Where K(t) is called the kernel and

29. Then a bit of calculation yields: And a bit more: And so we want:

30. The density estimate does not much depend on K(t), the kernel. But it does depend strongly on h, the bandwidth. We often choose a Gaussian (normal) kernel: Next we differentiate the integrated mean squared error and set it equal to 0 to find the optimal bandwidth (indept of x0). If we choose the Gaussian kernel and if f is normal then:

31. Recall, SE(median) = For NGC4382: n = 59, M = 26.974 Bootstrap result for NGC4382: SE(median) = .028 finite sample approx Final note: both bootstrap and density estimate are robust.

32. The median and the sign test (for testing H0: q = 0) are related through the L1 norm. To test H0: q = 0 we use S+(0) = # Xi > 0 which has a null binomial sampling distribution with parameters n and .5. This test is nonparametric and very robust.

33. Research Hypothesis: NGC4494 and NGC4382 differ in luminosity. Luminosity measurements (data) NGC 4494 (m = 101) 26.146 26.167 26.173…26.632 26.641 26.643 NGC 4382 (n = 59) 26.215 26.506 26.542…27.161 27.169 27.179 Statistical Model Two normal populations with possibly different means but with the same variance. Translation: H0: m4494= m4382 vs. H0: m4494m4382

34. Select a statistic: The two sample t statistic NGC 4494 (m = 101), NGC 4382 (n = 59) VERY STRANGE! The two sided t-test with significanc level .05 rejects the null hyp when |t| > 2. Recall that means and variances are not robust.

35. Table of true values of the significance level when the assumed level is .05. Another inconvenient truth: the true significance level can differ from .05 when some model assumptions fail.

36. An even more inconvenient truth: These problems extend to analysis of variance and regression. Seek alternative tests and estimates. We already have alternatives to the mean and t-test: the robust median and sign test.

37. We next consider nonparametric rank tests and estimates for comparing two samples. (Competes with the two sample t-test and difference in sample means.) Generally suppose: To test H0: D = 0 or to estimate D we introduce is the rank of Yj in the combined data.

38. The robust estimate of D is Provides the robustness Provides the comparison As opposed to which is not robust.

39. Research Hypothesis: NGC4494 and NGC4382 differ in luminosity. Luminosity measurements (data) X: NGC 4494 (n = 101) 26.146 26.167 26.173…26.632 26.641 26.643 Y: NGC 4382 (n = 59) 26.215 26.506 26.542…27.161 27.169 27.179 Statistical Model Two normal populations with possibly different medians but with the same scale. Translation: H0: D = 0vs. H0: D

40. Mann-Whitney Test and CI: NGC 4494, NGC 4382 N Median X: NGC 4494 101 26.659 Y: NGC 4382 59 26.974 Point estimate for Delta is 0.253 95.0 Percent CI for Delta is (0.182, 0.328) Mann-Whitney test: Test of Delta = 0 vs Delta not equal 0 is significant at 0.0000 (P-Value)

42. What to do about truncation. • See a statistician • Read the Johnson, Morrell, and Schick reference. and then • see a statistician. • Here is the problem: Suppose we want to estimate the difference in locations • between two populations: F(x) and G(y) = F(y – d). • But (with right truncation at a) the observations come from Suppose d > 0 and so we want to shift the X-sample to the right toward the truncation point. As we shift the Xs, some will pass the truncation point and will be eliminated from the data set. This changes the sample sizes and requires adjustment when computing the corresponding MWW to see if it is equal to its expectation. See the reference for details.

43. Comparison of NGC4382 and NGC 4494 Point estimate for d is .253 W = 6595.5 (sum of ranks of Ys) S+ = 4825.5 m = 101 and n = 59

44. Recall the two sample t-test is sensitive to the assumption of equal variances. The Mann-Whitney test is less sensitive to the assumption of equal scale parameters. The null distribution is nonparametric. It does not depend on the common underlying model distribution. It depends on the permutation principle: Under the null hypothesis, all (m+n)! permutations of the data are equally likely. This can be used to estimate the p-value of the test: sample the permutations, compute and store the MW statistics, then find the proportion greater than the observed MW.

45. Here’s a bad idea: Test the data for normality using, perhaps, the Kolmogorov-Smirnov test. If the test ‘accepts’ normality then use a t-test, and if it rejects normality then use a rank test. You can use the K-S test to reject normality. The inconvenient truth is that it may accept many possible models, some of which can be very disruptive to the t-test and sample means.

46. Absolute MagnitudePlanetary NebulaeMilky Way Abs Mag (n = 81) 17.537 15.845 15.449 12.710 15.499 16.450 14.695 14.878 15.350 12.909 12.873 13.278 15.591 14.550 16.078 15.438 14.741 …

48. But don’t be too quick to “accept” normality:

50. Null Hyp: Pop distribution, F(x) is normal The Kolmogorov-Smirnov Statistic The Anderson-Darling Statistic