methodology n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Methodology PowerPoint Presentation
Download Presentation
Methodology

Loading in 2 Seconds...

play fullscreen
1 / 48

Methodology - PowerPoint PPT Presentation


  • 103 Views
  • Uploaded on

Methodology. easy but important. ToC. What is performance evaluation about ? Metrics, Load and Goals Hidden Factors The Scientific Method Patterns. What is Performance Evaluation ?. Load. You need to define the load under which your system operates Make the difference between

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Methodology' - brigid


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
methodology

Methodology

easy but important

1

slide2
ToC
  • What is performance evaluation about ?
  • Metrics, Load and Goals
  • Hidden Factors
  • The Scientific Method
  • Patterns

2

slide4
Load
  • You need to define the load under which your system operates
  • Make the difference between
    • Intensity of the load (e.g. nb jobs per second)
    • Nature of the load
      • Statistical details that may matter: e.g. job sizes are heavy tailed or not
  • Benchmarks are artificial load generators; we will play with one of them
metric
Metric
  • Define a metric;examples
    • Response time
    • Power consumption
    • Throughput
  • Defineoperational conditions underwhichmetricismeasured (« Viewpoint », seeChapter 11)
know your goals
Know your goals
  • A1 and A3 are comparisons, A2 is an absolute statement
  • E2 is an engineering rule
3 hidden factors
3. Hidden Factors
  • Factor: an element that may impact the performance
    • (desired factors): intensity of load, number of servers
    • (nuisance factors): time of the day, presence of denial of service attack

13

why were we fooled
Why were we fooled ?
  • Hidden factor had a more important role than the factor we were interested in
  • We interpreted correlation as causality
  • Need to be aware of all factors and incorporate in the analysis
  • Or randomize experiment to reduce impact of hidden factors
simpson s paradox
Simpson’sParadox
  • A wellknownphenomenon -- Special case of Hidden Factor paradoxwhenmetricissuccess rate and factors are discrete
take home message
Take Home Message
  • Pitfallnumber 1 is the presence of hidden factor
  • Anystudyis susceptible to it
  • Easy for opponents to find
4 be scientific
4. Be Scientific
  • Joe measures performance of his Wireless Shop:
    • what would you conclude ?
scientific method
Scientific Method
  • Joe buys 2 more Access Points
    • improvement ?

Before

After

21

scientific method1
Scientific Method
  • A conclusion can only proven to be wrong
  • Do not draw conclusions unless the experiment was designed to test the statement
    • Measurement 1 suggested that the wireless network was congested, but the experiment was not designed to test this statement
  • Joe should: design an experiment to validate:H1: “the wireless network is the bottleneck”
    • for example: measure the number of collisions / packet loss
    • result: collision · 1%; conclusion: H1 is not valid
    • hypothesis H2: the server is saturated
    • experiment: measure memory utilization : result ¼ 100%

22

use of scientific method
Use of Scientific Method

TCP throughput may increase with mobility

(1) Duration of link failure period is impacted by speed(2) It has a negative impact on TCP throughput

Do more simulationsmeasure distrib of link failure period

Verify (1) and (2)

How ?

  • Recognize a fact
  • Pose a hypothesis
  • Verify the hypothesis on simulations / measurements designed to test it

28

take home message1
Take Home Message
  • You should not conclude from an experiment without trying to invalidate the conclusion
  • (Popper, 1934): you should alternate between the roles of
    • Proponent
    • Adversary
5 patterns
5. Patterns
  • These are common traits found in different situations
  • Knowing some of them may save a lot of time
bottlenecks may be your enemy
Bottlenecksmaybeyourenemy
  • Bottlenecks are like non invited people at a party – theymay impose their agenda
  • Previousexample: whatwe are measuringis the bottleneck, not the intended factor

33

bottlenecks are your friends1
Bottlenecks are YourFriends
  • Simplifyyour life, analyzebottlenecks !
  • In many cases, youmay ignore the rest

35

slide37

i

i’

i”

i”

37

slide38

i

i’

i”

i”

38

congestion collapse
Congestion Collapse
  • Definition: Offeredloadincreases, workdonedecreases
  • Frequent in complexsystems
  • May be due to
    • cost per job increaseswithload
    • Impatience
    • Rejection of jobs beforecompletion
  • Designer must do something to avoid congestion collapse
    • Eg. Admission control in Apache servers
    • Eg. TCP congestion control
  • Analyst must look for congestion collapse

40

competition side effect
Competition Side Effect
  • System balances resources according to some scheduling
  • Apparent paradox: put more resources, some get less

42

slide43

No TCP, users send as much can

  • Increase capacity of link 2 from 10 to 1000 kb/s

10 kb/s

43

competition side effect1
Competition Side Effect
  • Apparent paradox: put more resources, all get less

44

museum audio guide
Museum Audio Guide

Lowspeed USB connections atdocking station

High speed

45

latent congestion collapse
Latent Congestion Collapse
  • System is susceptible to congestion collapse
  • Low speed access prevents congestion collapse
  • Adding resources reveals congestion collapse

46

take home message2
Take Home Message
  • Watch for patterns, they are very frequent
    • Bottlenecks
    • Congestion collapse
    • Competition side effects
    • Latent Congestion collapse

47