cpe 619 selection of techniques and metrics l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
CPE 619 Selection of Techniques and Metrics PowerPoint Presentation
Download Presentation
CPE 619 Selection of Techniques and Metrics

Loading in 2 Seconds...

play fullscreen
1 / 38

CPE 619 Selection of Techniques and Metrics - PowerPoint PPT Presentation


  • 120 Views
  • Uploaded on

CPE 619 Selection of Techniques and Metrics. Aleksandar Milenković The LaCASA Laboratory Electrical and Computer Engineering Department The University of Alabama in Huntsville http://www.ece.uah.edu/~milenka http://www.ece.uah.edu/~lacasa. Overview.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'CPE 619 Selection of Techniques and Metrics' - ilyssa


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
cpe 619 selection of techniques and metrics

CPE 619Selection of Techniques and Metrics

Aleksandar Milenković

The LaCASA Laboratory

Electrical and Computer Engineering Department

The University of Alabama in Huntsville

http://www.ece.uah.edu/~milenka

http://www.ece.uah.edu/~lacasa

overview
Overview
  • One or more systems, real or hypothetical
  • You want to evaluate their performance
  • What technique do you choose?
    • Analytic Modeling?
    • Simulation?
    • Measurement?
  • What metrics do you use?
outline
Outline
  • Selecting an Evaluation Technique
  • Selecting Performance Metrics
    • Case Study
  • Commonly Used Performance Metrics
  • Setting Performance Requirements
    • Case Study
selecting an evaluation technique 1 of 4
Selecting an Evaluation Technique (1 of 4)
  • Which life-cycle stage the system is in?
    • Measurement only when something exists
    • If new, analytical modeling or simulation are only options
  • When are results needed? (often, yesterday!)
    • Analytic modeling only choice
    • Simulations and measurement can be same
      • But Murphy’s Law strikes measurement more often(“If anything can go wrong, it will.”)
  • What tools and skills are available?
    • Maybe languages to support simulation
    • Tools to support measurement (e.g.: packet sniffers, source code to add monitoring hooks)
    • Skills in analytic modeling (e.g.: queuing theory)
selecting an evaluation technique 2 of 4
Selecting an Evaluation Technique (2 of 4)
  • Level of accuracy desired?
    • Analytic modeling coarse (if it turns out to be accurate, even the analysts are surprised!)
    • Simulation has more details, but may abstract key system details
    • Measurement may sound real, but workload, configuration, etc., may still be missing
      • Accuracy can be high to none without proper design
    • Even with accurate data, still need to draw proper conclusions
      • E.g.: so response time is 10.2351 with 90% confidence. So what? What does it mean?
selecting an evaluation technique 3 of 4
Selecting an Evaluation Technique (3 of 4)
  • What are the alternatives?
    • Can explore trade-offs easiest with analytic models, simulations moderate, measurement most difficult
  • Cost?
    • Measurement generally most expensive
    • Analytic modeling cheapest (pencil and paper)
    • Simulation often cheap but some tools expensive
      • Traffic generators, network simulators
selecting an evaluation technique 4 of 4
Selecting an Evaluation Technique (4 of 4)
  • Saleability?
    • Much easier to convince people with measurements
    • Most people are skeptical of analytic modeling results since they are hard to understand
      • Often validate with simulation before using
  • Can use two or more techniques
    • Validate one with another
    • Most high-quality performance analysis papers have analytic model + simulation or measurement
summary table for evaluation technique selection
Summary Table for Evaluation Technique Selection

Criterion Modeling Simulation Measurement

1. Stage Any Any Prototype+

2. Time Small Medium Variesrequired

3. Tools Analysts Some Instrumentation

languages

4. Accuracy Low Moderate Varies

5. Trade-off Easy Moderate Difficultevaluation

6. Cost Small Medium High

7. Saleabilty Low Medium High

More important

Less important

outline9
Outline
  • Selecting an Evaluation Technique
  • Selecting Performance Metrics
    • Case Study
  • Commonly Used Performance Metrics
  • Setting Performance Requirements
    • Case Study
selecting performance metrics 1 of 3
Selecting Performance Metrics(1 of 3)

response timen. An unbounded, random variable … representing the

elapses between the time of sending a message and the time when the error

diagnostic is received. – S. Kelly-Bootle, The Devil’s DP Dictionary

Time

responsiveness

Possible Outcomes

Speed

Rate

productivity

Request

Resource

Correct

utilization

Done

Probability

System

Not

Correct

Errori

Reliability

Time

between

Not

Done

Eventk

Duration

Availability

Time

between

selecting performance metrics 2 of 3
Selecting Performance Metrics(2 of 3)
  • Mean is what usually matters
    • But do not overlook the effect of variability
  • Individual vs. Global (systems shared by many users)
    • May be at odds
    • Increase individual may decrease global
      • E.g.: response time at the cost of throughput
    • Increase global may not be most fair
      • E.g.: throughput of cross traffic
  • Performance optimizations of bottleneck have most impact
    • E.g.: Response time of Web request
    • Client processing 1s, Latency 500ms, Server processing 10s  Total is 11.5 s
    • Improve client 50%?  11 s
    • Improve server 50%?  6.5 s
selecting performance metrics 3 of 3
Selecting Performance Metrics(3 of 3)
  • May be more than one set of metrics
    • Resources: Queue size, CPU Utilization, Memory Use …
  • Criteria for selecting subset, choose:
    • Low variability – need fewer repetitions
    • Non redundancy – don’t use 2 if 1 will do
      • E.g.: queue size and delay may provide identical information
    • Completeness – should capture tradeoffs
      • E.g.: one disk may be faster but may return more errors so add reliability measure
outline13
Outline
  • Selecting an Evaluation Technique
  • Selecting Performance Metrics
    • Case Study
  • Commonly Used Performance Metrics
  • Setting Performance Requirements
    • Case Study
case study 1 of 5
Case Study (1 of 5)
  • Computer system of end-hosts sending packets through routers
    • Congestion occurs when number of packets at router exceed buffering capacity
  • Goal: compare two congestion control algorithms
  • User sends block of packets to destination; Four possible outcomes:
    • A) Some delivered in order
    • B) Some delivered out of order
    • C) Some delivered more than once
    • D) Some dropped
case study 2 of 5
Case Study (2 of 5)
  • For A), straightforward metrics exist:

1) Response time: delay for individual packet

2) Throughput: number of packets per unit time

3) Processor time per packet at source

4) Processor time per packet at destination

5) Processor time per packet at router

  • Since large response times can cause extra (unnecessary) retransmissions:

6) Variability in response time (is also important)

case study 3 of 5
Case Study (3 of 5)
  • For B), out-of-order packets cannot be delivered to the user immediately
    • They are often discarded (considered dropped)
    • Alternatively, they are stored in destination buffers awaiting arrival of intervening packets

7) Probability of out of order arrivals

  • For C), consume resources without any use

8) Probability of duplicate packets

  • For D), for many reasons is undesirable

9) Probability of lost packets

  • Also, excessive loss can cause disconnection

10) Probability of disconnect

case study 4 of 5
Case Study (4 of 5)
  • Since a multi-user system, want fairness:
    • 11) Fairness = A function of variability of throughput across users; for any given set of user throughputs (x1, x2, …, xn), the fairness is:

f(x1, x2, …, xn) = (xi)2 / (n xi2)

    • Index between 0 and 1
      • All users get same, then 1
      • If k users get equal throughput and n-k get zero, than index is k/n
case study 5 of 5
Case Study (5 of 5)
  • After a few experiments (pilot tests)
    • Found throughput and delay redundant
      • higher throughput had higher delay
      • instead, combine with power = thrput/delay
    • Found variance in response time redundant with probability of duplication and probability of disconnection
      • Drop variance in response time
  • Thus, left with nine metrics
outline19
Outline
  • Selecting an Evaluation Technique
  • Selecting Performance Metrics
    • Case Study
  • Commonly Used Performance Metrics
  • Setting Performance Requirements
    • Case Study
commonly used performance metrics
Commonly Used Performance Metrics
  • Response Time
    • Turn around time
    • Reaction time
    • Stretch factor
  • Throughput
    • Operations/second
    • Capacity
    • Efficiency
    • Utilization
  • Reliability
    • Uptime
    • MTTF
response time 1 of 2

User’s

Request

System’s

Response

Time

Response Time (1 of 2)
  • Interval between user’s request and system response
  • But simplistic since requests and responses are not instantaneous
    • Users spend time typing the request and the system takes time to output the response
response time 2 of 2
Response Time (2 of 2)

System

Starts

Execution

System

Starts

Response

User

Starts

Request

User

Finishes

Request

System

Finishes

Response

  • Can have two measures of response time
    • Both ok, but 2 preferred if execution long
  • Think time can determine system load

Time

Reaction

Time

Think

Time

Response

Time 1

Response

Time 2

response time
Response Time+
  • Turnaround time – time between submission of a job and completion of output
    • For batch job systems
  • Reaction time - Time between submission of a request and beginning of execution
    • Usually need to measure inside system since nothing externally visible
  • Stretch factor – ratio of response time at a particular load to the response time at minimum load
    • Most systems have higher response time as load increases
throughput 1 of 2
Throughput (1 of 2)
  • Rate at which requests can be serviced by system (requests per unit time)
    • Batch: jobs per second
    • Interactive: requests per second
    • CPUs
      • Millions of Instructions Per Second (MIPS)
      • Millions of Floating-Point Ops per Sec (MFLOPS)
    • Networks: pkts per second or bits per second
    • Transactions processing: Transactions Per Second (TPS)
throughput 2 of 2

Knee

Nominal

Capacity

Thrput

Usable

Capacity

Knee

Capacity

Load

Response

Time

Load

Throughput (2 of 2)
  • Nominal capacity is ideal (e.g.: 10 Mbps)
  • Usable capacity is achievable (e.g.: 9.8 Mbps)
  • Knee is where response time goes up rapidly for small increase in throughput
  • Throughput increases as load increases, to a point
efficiency

Efficiency

Number of Processors

Efficiency
  • Ratio of maximum achievable throughput (e.g.: 9.8 Mbps) to nominal capacity (e.g.: 10 Mbps)  98%
  • For multiprocessor, ratio of n-processor to that of one-processor (in MIPS or MFLOPS)
utilization
Utilization
  • Typically, fraction of time resource is busy serving requests
    • Time not being used is idle time
    • System managers often want to balance resources to have same utilization
      • E.g.: equal load on CPUs
      • But may not be possible. E.g.: CPU when I/O is bottleneck
  • May not be time
    • Processors – busy / total makes sense
    • Memory – fraction used / total makes sense
miscellaneous metrics
Miscellaneous Metrics
  • Reliability
    • Probability of errors or mean time between errors (error-free seconds)
  • Availability
    • Fraction of time system is available to service requests (fraction not available is downtime)
    • Mean Time To Failure (MTTF) is mean uptime
      • Useful, since availability high (downtime small) may still be frequent and no good for long request
  • Cost/Performance ratio
    • Total cost / Throughput, for comparing 2 systems
    • Ex: For Transaction Processing system may want Dollars / TPS
utility classification

HB

Better

Utility

LB

NB

Metric

Better

Utility

Utility

Best

Metric

Metric

Utility Classification
  • HB – Higher is better (ex: throughput)
  • LB - Lower is better (ex: response time)
  • NB – Nominal is best (ex: utilization)
outline30
Outline
  • Selecting an Evaluation Technique
  • Selecting Performance Metrics
    • Case Study
  • Commonly Used Performance Metrics
  • Setting Performance Requirements
    • Case Study
setting performance requirements 1 of 2
Setting Performance Requirements(1 of 2)
  • Consider these typical requirement statements
    • The system should be both processing and memory efficient. It should not create excessive overhead
    • There should be an extremely low probability that the network will duplicate a packet, deliver it to a wrong destination, or change the data
  • What’s wrong?
setting performance requirements 2 of 2
Setting Performance Requirements(2 of 2)
  • General Problems
    • Nonspecific – no numbers. Only qualitative words (rare, low, high, extremely small)
    • Nonmeasureable – no way to measure and verify that the system meets requirements
    • Nonacceptable – numerical values of requirements are set based upon what can be achieved or on what looks good; If set on what can be achieved, they may turn out to be too low
    • Nonrealizable – numbers based on what sounds good, but once started are too high
    • Nonthorough – no attempt is made to specify all outcomes
outline33
Outline
  • Selecting an Evaluation Technique
  • Selecting Performance Metrics
    • Case Study
  • Commonly Used Performance Metrics
  • Setting Performance Requirements
    • Case Study
setting performance requirements case study 1 of 2
Setting Performance Requirements: Case Study (1 of 2)
  • Performance for high-speed LAN
  • Speed – if packet delivered, time taken to do so is important

A) Access delay should be less than 1 sec

B) Sustained throughput at least 80 Mb/s

  • Reliability

A) Prob of bit error less than 10-7

B) Prob of frame error less than 1%

C) Prob of frame error not caught 10-15

D) Prob of frame miss-delivered due to uncaught error 10-18

E) Prob of duplicate 10-5

F) Prob of losing frame less than 1%

setting performance requirements case study 2 of 2
Setting Performance Requirements: Case Study (2 of 2)
  • Availability

A) Mean time to initialize LAN < 15 msec

B) Mean time between LAN inits > 1 minute

C) Mean time to repair < 1 hour

D) Mean time between LAN partitions > ½ week

  • All above values were checked for realizeability by modeling, showing that LAN systems satisfying the requirements were possible
part i things to remember
Part I: Things to Remember
  • Systematic Approach
    • Define the system, list its services, metrics, parameters, decide factors, evaluation technique, workload, experimental design, analyze the data, and present results
  • Selecting Evaluation Technique
    • The life-cycle stage is the key. Other considerations are: time available, tools available, accuracy required, trade-offs to be evaluated, cost, and saleability of results.
part i things to remember37
Part I: Things to Remember
  • Selecting Metrics
    • For each service list time, rate, and resource consumption
    • For each undesirable outcome, measure the frequency and duration of the outcome
    • Check for low-variability, non-redundancy, and completeness.
  • Performance requirements:
    • Should be SMART. Specific, measurable, acceptable, realizable, and thorough.
homework 1
Homework #1
  • Read Chapters #1, #2, #3
  • Submit answers to exercises
    • 2.2 (assume the system is personal computer)
    • 3.1 and 3.2
  • Due: Monday, August 31, 2009
  • Submit by email to instructor with subject “CPE619-HW1”