Methodology

1 / 48

# Methodology - PowerPoint PPT Presentation

Methodology. easy but important. ToC. What is performance evaluation about ? Metrics, Load and Goals Hidden Factors The Scientific Method Patterns. What is Performance Evaluation ?. Load. You need to define the load under which your system operates Make the difference between

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

## Methodology

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

### Methodology

easy but important

1

ToC
• What is performance evaluation about ?
• Hidden Factors
• The Scientific Method
• Patterns

2

• You need to define the load under which your system operates
• Make the difference between
• Intensity of the load (e.g. nb jobs per second)
• Statistical details that may matter: e.g. job sizes are heavy tailed or not
• Benchmarks are artificial load generators; we will play with one of them
Metric
• Define a metric;examples
• Response time
• Power consumption
• Throughput
• Defineoperational conditions underwhichmetricismeasured (« Viewpoint », seeChapter 11)
• A1 and A3 are comparisons, A2 is an absolute statement
• E2 is an engineering rule
3. Hidden Factors
• Factor: an element that may impact the performance
• (desired factors): intensity of load, number of servers
• (nuisance factors): time of the day, presence of denial of service attack

13

Why were we fooled ?
• Hidden factor had a more important role than the factor we were interested in
• We interpreted correlation as causality
• Need to be aware of all factors and incorporate in the analysis
• Or randomize experiment to reduce impact of hidden factors
• A wellknownphenomenon -- Special case of Hidden Factor paradoxwhenmetricissuccess rate and factors are discrete
Take Home Message
• Pitfallnumber 1 is the presence of hidden factor
• Anystudyis susceptible to it
• Easy for opponents to find
4. Be Scientific
• Joe measures performance of his Wireless Shop:
• what would you conclude ?
Scientific Method
• Joe buys 2 more Access Points
• improvement ?

Before

After

21

Scientific Method
• A conclusion can only proven to be wrong
• Do not draw conclusions unless the experiment was designed to test the statement
• Measurement 1 suggested that the wireless network was congested, but the experiment was not designed to test this statement
• Joe should: design an experiment to validate:H1: “the wireless network is the bottleneck”
• for example: measure the number of collisions / packet loss
• result: collision · 1%; conclusion: H1 is not valid
• hypothesis H2: the server is saturated
• experiment: measure memory utilization : result ¼ 100%

22

Use of Scientific Method

TCP throughput may increase with mobility

(1) Duration of link failure period is impacted by speed(2) It has a negative impact on TCP throughput

Do more simulationsmeasure distrib of link failure period

Verify (1) and (2)

How ?

• Recognize a fact
• Pose a hypothesis
• Verify the hypothesis on simulations / measurements designed to test it

28

Take Home Message
• You should not conclude from an experiment without trying to invalidate the conclusion
• (Popper, 1934): you should alternate between the roles of
• Proponent
5. Patterns
• These are common traits found in different situations
• Knowing some of them may save a lot of time
Bottlenecksmaybeyourenemy
• Bottlenecks are like non invited people at a party – theymay impose their agenda
• Previousexample: whatwe are measuringis the bottleneck, not the intended factor

33

Bottlenecks are YourFriends
• In many cases, youmay ignore the rest

35

i

i’

i”

i”

37

i

i’

i”

i”

38

Congestion Collapse
• Frequent in complexsystems
• May be due to
• Impatience
• Rejection of jobs beforecompletion
• Designer must do something to avoid congestion collapse
• Eg. Admission control in Apache servers
• Eg. TCP congestion control
• Analyst must look for congestion collapse

40

Competition Side Effect
• System balances resources according to some scheduling
• Apparent paradox: put more resources, some get less

42

No TCP, users send as much can

• Increase capacity of link 2 from 10 to 1000 kb/s

10 kb/s

43

Competition Side Effect
• Apparent paradox: put more resources, all get less

44

Museum Audio Guide

Lowspeed USB connections atdocking station

High speed

45

Latent Congestion Collapse
• System is susceptible to congestion collapse
• Low speed access prevents congestion collapse
• Adding resources reveals congestion collapse

46

Take Home Message
• Watch for patterns, they are very frequent
• Bottlenecks
• Congestion collapse
• Competition side effects
• Latent Congestion collapse

47