methodology n.
Skip this Video
Loading SlideShow in 5 Seconds..
Methodology PowerPoint Presentation
Download Presentation

Loading in 2 Seconds...

play fullscreen
1 / 48

Methodology - PowerPoint PPT Presentation

  • Uploaded on

Methodology. easy but important. ToC. What is performance evaluation about ? Metrics, Load and Goals Hidden Factors The Scientific Method Patterns. What is Performance Evaluation ?. Load. You need to define the load under which your system operates Make the difference between

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript


easy but important


  • What is performance evaluation about ?
  • Metrics, Load and Goals
  • Hidden Factors
  • The Scientific Method
  • Patterns


  • You need to define the load under which your system operates
  • Make the difference between
    • Intensity of the load (e.g. nb jobs per second)
    • Nature of the load
      • Statistical details that may matter: e.g. job sizes are heavy tailed or not
  • Benchmarks are artificial load generators; we will play with one of them
  • Define a metric;examples
    • Response time
    • Power consumption
    • Throughput
  • Defineoperational conditions underwhichmetricismeasured (« Viewpoint », seeChapter 11)
know your goals
Know your goals
  • A1 and A3 are comparisons, A2 is an absolute statement
  • E2 is an engineering rule
3 hidden factors
3. Hidden Factors
  • Factor: an element that may impact the performance
    • (desired factors): intensity of load, number of servers
    • (nuisance factors): time of the day, presence of denial of service attack


why were we fooled
Why were we fooled ?
  • Hidden factor had a more important role than the factor we were interested in
  • We interpreted correlation as causality
  • Need to be aware of all factors and incorporate in the analysis
  • Or randomize experiment to reduce impact of hidden factors
simpson s paradox
  • A wellknownphenomenon -- Special case of Hidden Factor paradoxwhenmetricissuccess rate and factors are discrete
take home message
Take Home Message
  • Pitfallnumber 1 is the presence of hidden factor
  • Anystudyis susceptible to it
  • Easy for opponents to find
4 be scientific
4. Be Scientific
  • Joe measures performance of his Wireless Shop:
    • what would you conclude ?
scientific method
Scientific Method
  • Joe buys 2 more Access Points
    • improvement ?




scientific method1
Scientific Method
  • A conclusion can only proven to be wrong
  • Do not draw conclusions unless the experiment was designed to test the statement
    • Measurement 1 suggested that the wireless network was congested, but the experiment was not designed to test this statement
  • Joe should: design an experiment to validate:H1: “the wireless network is the bottleneck”
    • for example: measure the number of collisions / packet loss
    • result: collision · 1%; conclusion: H1 is not valid
    • hypothesis H2: the server is saturated
    • experiment: measure memory utilization : result ¼ 100%


use of scientific method
Use of Scientific Method

TCP throughput may increase with mobility

(1) Duration of link failure period is impacted by speed(2) It has a negative impact on TCP throughput

Do more simulationsmeasure distrib of link failure period

Verify (1) and (2)

How ?

  • Recognize a fact
  • Pose a hypothesis
  • Verify the hypothesis on simulations / measurements designed to test it


take home message1
Take Home Message
  • You should not conclude from an experiment without trying to invalidate the conclusion
  • (Popper, 1934): you should alternate between the roles of
    • Proponent
    • Adversary
5 patterns
5. Patterns
  • These are common traits found in different situations
  • Knowing some of them may save a lot of time
bottlenecks may be your enemy
  • Bottlenecks are like non invited people at a party – theymay impose their agenda
  • Previousexample: whatwe are measuringis the bottleneck, not the intended factor


bottlenecks are your friends1
Bottlenecks are YourFriends
  • Simplifyyour life, analyzebottlenecks !
  • In many cases, youmay ignore the rest














congestion collapse
Congestion Collapse
  • Definition: Offeredloadincreases, workdonedecreases
  • Frequent in complexsystems
  • May be due to
    • cost per job increaseswithload
    • Impatience
    • Rejection of jobs beforecompletion
  • Designer must do something to avoid congestion collapse
    • Eg. Admission control in Apache servers
    • Eg. TCP congestion control
  • Analyst must look for congestion collapse


competition side effect
Competition Side Effect
  • System balances resources according to some scheduling
  • Apparent paradox: put more resources, some get less



No TCP, users send as much can

  • Increase capacity of link 2 from 10 to 1000 kb/s

10 kb/s


competition side effect1
Competition Side Effect
  • Apparent paradox: put more resources, all get less


museum audio guide
Museum Audio Guide

Lowspeed USB connections atdocking station

High speed


latent congestion collapse
Latent Congestion Collapse
  • System is susceptible to congestion collapse
  • Low speed access prevents congestion collapse
  • Adding resources reveals congestion collapse


take home message2
Take Home Message
  • Watch for patterns, they are very frequent
    • Bottlenecks
    • Congestion collapse
    • Competition side effects
    • Latent Congestion collapse