Did Something Change? Using Statistical Techniques to Interpret Service and Resource Metrics

1 / 60

# Did Something Change? Using Statistical Techniques to Interpret Service and Resource Metrics - PowerPoint PPT Presentation

Did Something Change? Using Statistical Techniques to Interpret Service and Resource Metrics. Frank Bereznay. Did Something Change?

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## Did Something Change? Using Statistical Techniques to Interpret Service and Resource Metrics

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Did Something Change?Using Statistical Techniques to Interpret Service and Resource Metrics

Frank Bereznay

Did Something Change?

In a perfect world, one would always know the answer to that question. Unfortunately, nobody works in a perfect world. This paper / presentation will explore statistical techniques used to look for deviations in metrics that are due to assignable causes as opposed to the period to period variation that is normally present. Hypothesis Testing, Statistical Process Control, Multivariate Adaptive Statistical Filtering, and Analysis of Variance will be compared and contrasted. SAS code will be used to perform the analysis. Exploratory analysis techniques will be used to build populations for analysis purposes.

Abstract

It’s the population that counts

Repeatable processes

Four techniques to review

Hypothesis Testing

Statistical Process Control

Analysis of Variance (ANOVA)

Example

Summary & Questions

Outline
It is the Population that Counts.

Populations have Parameters.

Samples have Statistics.

The Science of Statistics is all about estimating Population Parameters by taking Samples and calculating Statistics.

It can be anything you want it to be, but

It must have well defined boundaries.

A production cycle of a manufacturing process.

A work shift.

A bottling run for a particular wine vintage.

It must be randomly sampled.

Standard topic for first year Stat Class.

Simple and easy to do.

Interpretation of results has been misunderstood.

Hypothesis Testing
Start with a statement you wish to contradict or disprove. Typically this is the status quo. It becomes the null hypothesis.

The average message rate is 15 per minute.

Create an alternative hypothesis that contradicts the null hypothesis.

The average message rate is not 15 per minute.

Hypothesis Testing
What is the population we are working with here?

It is a 24 hour period.

We must randomly sample across the entire period.

We randomly collect message rates at 10 different points in time.

13,14,16,11,16,15,12,16,12,14

Hypothesis Testing
Hypothesis Testing

Calculation of the t statistic

with 9 (N-1) Degrees of Freedom

So, What does this tell us?

The official statement is:

At a 95% confidence level, the data is insufficient for us to state the mean of the population is not 15 for the 24 hour period being examined.

Important point, the contrary is not necessarily true.

This does not prove in any way the population mean is 15.

Hypothesis Testing
Statistical Assumptions that need to be considered.

Underlying population does not need to be normally distributed.

The population must be randomly sampled.

Hypothesis Testing
Some practical uses.

Validating we have met an SLA.

Looking to see if something is not what we expect it to be.

Key point

This technique combines an a priori expectation about a quality metric with sampled data.

You need to know your data and choose wisely.

Hypothesis Testing
Two Legends standout in this area:

Walter Shewhart

W. Edwards Deming

SPC is conceptually similar to Hypothesis Testing, but computationally different.

No a priori data point is needed.

Data is sub-grouped for calculation purposes.

SPC and Hypothesis Testing can produce different results for the same set of data.

Statistical Process Control
Statistical Process Control

Sample Order Output

Statistical Assumptions that need to be considered.

The data does not need to be normally distributed.

Proper sub grouping of the data is fundamental to the technique.

Sampling plan must be random and cover the boundaries of the population being examined.

Statistical Process Control
Practical Uses

Useful for measuring discrete physical objects.

Things that have physical properties.

Counts for outputs.

Dollar volumes for orders / sales.

Not appropriate for interval based instrumentation data we frequently use.

Statistical Process Control
Developed by Annie Shum and Jeff Buzen.

Subject of 1995 CMG Paper by same name.

Practitioner’s approach to create a statistical detection technique which addresses the unique challenges of the interval driven time series datasets used by Computer Resource Management Professionals.

Variance based statistical detection techniques are based on repeatable processes.

Filling a bottle with wine.

Manufacturing a roll of paper.

Commercial computer workloads are generally not repeatable processes (and that is an understatement!).

Why MASF?
A two step process is established:

A Reference Set is created during a period of normal operation in place of a random sample.

The Reference Set is used as a set of criteria to examine data from subsequent periods.

MASF
What is a normal period?

Workloads vary by time of day, day of week and month of year.

MASF – Reference Set
The collected data can / should be grouped into set of hours with same characteristics.

Increases number of samples per collection period.

Hour

Day

8

9

10

11

12

13

14

15

16

17

Mon

1

2

2

2

3

4

4

4

5

5

Tue

6

7

7

7

3

8

8

8

9

9

Wed

6

7

7

7

3

8

8

8

9

9

Thur

6

7

7

7

3

8

8

8

9

9

Fri

10

11

11

11

12

13

13

13

14

14

MASF – Aggregation Policies
MASF – Detection Limits

Monday

Tuesday thru Thursday

Friday

Very robust statistical detection technique.

More of a framework than a specific procedure.

Reference set is user defined.

Measurement methodology is user defined.

MASF - Summary

Ideally 10 to 20 points per reference set.

Longer term datasets subject to Time Series influences which distorts metrics.

This technique should be included in every Resource Management Specialist’s toolkit!

MASF Summary
A comparison of parameters across populations.

Best explained by why it was developed.

Agricultural work in the late 1800’s to improve crop yields.

Plot of land was divided into multiple areas and subjected to different treatments.

Test was developed to compare the effects of these different treatments on crop yield.

Analysis of Variance (ANOVA)
ANOVA
• Example of how this type of experiment would be setup:
• Important Point - We are dealing with six separate populations.
Same ground rules as Hypothesis Testing

Start of by assuming all population means are equal.

Null Hypothesis

Attempt to prove they are not all the same.

Alternative Hypothesis

However, calculation of the result is very laborious and best done by a computer.

ANOVA
ANOVA
• Test stated in similar fashion to Hypothesis Testing.

Can’t prove any mean is different – end of test.

Accepting Alternative Hypothesis has an interesting twist.

One or more of the means are different – but which one(s) is/are different?

ANOVA
Turkey test answers the Alternative Hypothesis question.

John Tukey developed a technique to group means of an ANOVA test when the Alternative Hypothesis is accepted.

We now have a way to take a set of multiple data populations and segment them into like groups.

ANOVA
ANOVA
• SMF Data Volume Example
SAS Proc ANOVA Procedure

ProcANOVA;

Class Day;

Model Count = Day;

Means Day / Tukey;

Run;

ANOVA
Key Results from Test

Pr > F = .0424

We conclude at a 95% confidence level that one or more of the means are different.

Tukey Test

Monday and Friday are different

All other days are the same

A certain degree of ambiguity

ANOVA
Typical way to report or display results of Tukey test.

Mon Tue Wed Thur Fri

|-------------------------------|

|----------------------------|

ANOVA
Second test.

Compare the day of the week across weeks

ProcANOVA;

Format Date Date8.;

Class Date;

Model Count = Date; By Day;

Run;

ANOVA
Results from second test.

All five tests accepted the null hypothesis.

Pr > F were all in the high 90% range.

So the ‘official’ statement is’

The data is insufficient to conclude there is any difference in the mean value for a day of the week across weeks.

ANOVA
Statistical Assumptions that need to be considered.

Sufficient data is needed to obtain 6 to 10 observations for each treatment.

Need to be sensitive to correlated data.

Sampling plan must be random and cover the boundaries of the population being examined.

ANOVA
Practical Uses

Comparing data from multiple days to see if it is the same or different.

Use it as a clustering technique to build aggregated data groups for a MASF analysis.

Multiple factor ANOVAs can look at multiple treatments (factors) at the same time.

Day of week and hour of day.

A very powerful tool that should be in everybody’s toolkit!

ANOVA
One Month of Prime Shift usage data for an OLTP server.

The MASF technique will be used to look for deviations.

The first three weeks will be used to be the reference set to examine the fourth weeks data.

ANOVA will be used to create Aggregation Policies to cluster the hourly data.

Midrange Server Example
ANOVA test was performed on the hours of the day.

Two overlapping groups were identified.

CPUAVE 33.1 30.8 30.7 30.5 29.7 29.6 29.4 28.8

Hour 8 11 12 9 10 13 14 15

|----------------------|

|--------------------------------------------|

Midrange Server Example
A second ANOVA test was performed on the day of the week.

Identified two non-overlapping groups.

Group 1

Monday and Friday.

Group 2

Tuesday, Wednesday and Thursday.

Midrange Server Example
Midrange Server Example
• The following aggregation policy was built for this workload.
Midrange Server Example
• The aggregation policy was used to build the following reference set from the table of hour usage metrics:
Midrange Server Example
• Plotting this along with the actual data from the fourth week produced the following control chart for Monday:
So, What is in your toolkit?

Pick up these tools at your nearest CMG meeting. They do take some getting used to, but are worth the learning curve.

Hypothesis Testing, Statistical Process Control, MASF and ANOVA

Be very wary of your data.

The Time Series Data we routinely work with is a very complicated multi-dimensional dataset.

Get to know you data. The better you know the data, the better you know your workload.

Summary

I. Trubin’s CMG papers on application of MASF and variance based statistical detection techniques.

2001 – Exception Detection System, Based on Statistical Process Control Concept.

2002 – Global and Application Levels Exception Detection System, Based on MASF Technique

2003 – Disk Subsystem Capacity Management, Based on Business Drivers, I/O Performance Metrics and MASF

2004 – Mainframe Global and Workload Levels Statistical Exception Detection System, Based on MASF

2005 – Capturing Workload Pathology by Statistical Exception Detection System.

Summary