Framework usages metrics proposal for tgt
This presentation is the property of its rightful owner.
Sponsored Links
1 / 17

Framework, Usages, Metrics Proposal for TGt PowerPoint PPT Presentation


  • 64 Views
  • Uploaded on
  • Presentation posted in: General

Framework, Usages, Metrics Proposal for TGt. Pratik Mehta, Fahd Pirzada – Dell Paul Canaan – Intel Amer Hassan, Don Berry – Microsoft September 2004. Excerpt from Nov 2003 Presentation – Schedule Plan.

Download Presentation

Framework, Usages, Metrics Proposal for TGt

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Framework usages metrics proposal for tgt

Framework, Usages, Metrics Proposal for TGt

Pratik Mehta, Fahd Pirzada – Dell

Paul Canaan – Intel

Amer Hassan, Don Berry – Microsoft

September 2004


Excerpt from nov 2003 presentation schedule plan

Excerpt from Nov 2003 Presentation – Schedule Plan

The milestones set out in Nov 2003 are on track … WPP/TGt have been meeting expectations …

Presenter: Stephen Berger, Reference doc.: 11-03-0929-00-0wng


Excerpt from nov 2003 presentation schedule plan1

Excerpt from Nov 2003 Presentation – Schedule Plan

The milestones set out in Nov 2003 are on track … WPP/TGt have been meeting expectations …

Presenter: Stephen Berger, Reference doc.: 11-03-0929-00-0wng


Excerpt from nov 2003 presentation scope

Excerpt from Nov 2003 Presentation – Scope

WPP/TGt continues to focus on the scope set out in Nov 2003 …

Proposed Scope

Presenter: Pratik Mehta, Reference doc.: 11-03-0927-00-0wng


Excerpt from nov 2003 presentation metrics

Excerpt from Nov 2003 Presentation – Metrics

WPP/TGt continues to discuss metrics highlighted in Nov 2003 by Broadcom…

  • Deciding what parameters are to be considered is the challenge.

  • How do we transform user perception of performance into a set of repeatably-measurable quantities?

    • Throughput and Range

    • Visibility of APs

    • Delays in association

    • Host CPU utilization

    • Ability to roam without loss of connections

    • Etc.

Slides 16-20 of that presentation delve further into metrics that are still relevant …

… TGt can decide what makes sense …

Presenter: Jason Trachewsky et al, Reference doc.: 11-03-0933-00-0wng


Excerpts from presentations metrics

Excerpts from Presentations – Metrics

WPP/TGt continues to discuss usages metrics…

  • Roaming is important to consider

  • Roaming brings in many variables and causes that impact performance

Presenter: Bob Mandeville, Reference doc.: 11-03-0617-00-0wng

  • More recent presentation on metrics by Bob …

    • Maximum Forwarding Rate

    • FWMOL (Fwd rate at max offered load)

    • Frame Loss, Frame Loss Rate

    • Latency

    • Jitter

    • Association Capacity, Association Rate

    • Rate versus Range

Presenter: Bob Mandeville, Reference doc.: 11-04-0987-00-0wpp


Excerpt from recent wpp tgt presentation framework

Excerpt from Recent WPP/TGt Presentation – Framework

WPP/TGt has discussed a framework approach … below is as highlighted by Intel…

Framework for WPP

  • Test Environment

    • Come to consensus on environment types (NLOS, LOS, Conductive, etc)

    • Develop a mechanism to characterize the environment

    • Propose recommended practices for the environmental set up

  • Metrics

    • Define metrics relevant to characterize the environment

    • Identify metrics that impact WLAN performance as well as corresponding methodology to obtain these metrics

    • Devise a mechanism to predict the metric performance

  • Test Case Template

    • Come to consensus on the necessary metrics to characterize the environment that should be reported in a test template

    • Define test case scenarios for each metric

    • Determine recommended practice for reporting/analyzing of the results

  • Usage

    • Come to consensus on usage scenarios that represent the WLAN traffic pattern (file transfers, streaming media, connectivity, voice over IP, etc)

    • Categorize which metrics have the greatest impact on the performance of those usage scenarios

Presenter: Paul C. Canaan, Reference doc.: 11-04-0802-01-0wpp


Interim summary

Interim Summary

  • TGt is focusing on the right approaches

  • TGt is also meeting deliveries that were highlighted 9 months ago

  • TGt has the right constituencies and experts represented

  • … In short, TGt is a healthy group, and can move forward and deliver

  • There is a call for urgency in the short-term to address some key industry needs

    • OEM needs during product development

    • Reviewer/Publisher needs for product reviews

    • Guidelines for system integrators

  • The following slides provide a proposal to move forward


Restatement of tgt framework

Restatement of TGt Framework

Usage + Environment  Metrics  Methodology  Test Template …  Prediction

Usage

+

Environment

Metrics

Sub-metrics

Methodology

Test Environment

Test Procedure

Test Template

Use Measurements

To Predict


Definition of terms in tgt framework

Definition of Terms in TGt Framework

Usage + Environment  Metrics  Methodology  Test Template …  Prediction

Usage

+

Environment

  • Usage and environment of end-user

Metrics

Sub-metrics

  • Metrics are the key performance indicators as perceived by users – for the usages they employ in their environments (above)

  • Sub-metrics are drivers of those metrics

  • Sub-metrics help understand what affects the metrics

Methodology

Test Environment

Test Procedure

Test Template

  • Methodology is the way to measure metrics and sub-metrics

  • Test Environment is a model of the user’s environment

  • Test Procedure is the set of steps to perform measurements

  • Test Template is the content and format used to report the measurements

Use Measurements

To Predict

  • Output from Methodology and Measurements are used to predict performance


Proposal focus on three cases

Proposal – Focus on three Cases

Usage + Environment  Metrics  Methodology  …

  • 1. Data Oriented Applications + Home, Corporate Environment

  • Metrics:

  • Throughput, Range, Directionality

  • Sub-metrics:

  • Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power

  • Methodology:

  • Chariot scripts – Chamber, Indoor

  • Web-downloads – Indoor

  • File-Transfers, File-Sharing – Indoor

  • Rotating tables

  • 3. Latency Sensitive Apps + Home, Corporate Environment

  • Metrics:

  • All of 1, Latency, Jitter

  • Sub-Metrics:

  • MOS/PESQ, PSQM, One-way delay, Packet loss

  • Methodology:

  • Decoder/Encoder characteristics, Network protocols (H.323, P2P, SIP)

  • Apps: Netmeeting, Skype, Vonage

  • 2. Streaming Media Apps + Home, Corporate Environment

  • Metrics:

  • All of 1, Frames dropped, Image quality

  • Sub-Metrics:

  • TCP Re-transmissions, Packet errors, Latency, AV synchronization, Control Channel (Gaming, Multicast), Image quality (Jitter, Blockiness, Blur), Bandwidth stability, Buffering (pre-roll buffer, jitter buffer)

  • Methodology:

  • Unicast, Multicast, Real-time streaming, Stored content, Decoder/Encoder characteristics, Network protocols (TCP, UDP), Buffering

  • Number of streams,

  • Apps: Bobsled, VideoLAN, Real server.

Each of these Usage cases will need to be partitioned … (next slide shows example for #1)

Attributes common to all three usages to consider:

No. of Clients; No. of APs; Stationary/Roaming; Encryption/Authentication


Partitioning approach for usage case 1

Partitioning Approach for Usage Case #1

Usage + Environment  Metrics  Methodology  Test Template …

1. Data Oriented Applications + Home, Corporate Environment

  • Chariot Throughput + Indoor, Chamber Environment

  • Metrics:

  • Throughput, Range, Directionality

  • Sub-Metrics

  • Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power

  • Methodology: ?

  • Web Downloads + Indoor Environment

  • Metrics:

  • Latency, Throughput, Range

  • Sub-Metrics

  • Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power

  • Methodology: ?

  • File transfer, File sharing + Indoor Environment

  • Metrics:

  • Throughput, Range

  • Sub-Metrics

  • Path loss, EVM measurements, Interference, Multipath, Noise, SNR, Receive sensitivity, Transmit power

  • Methodology: ?


Similar partitioning is needed for 2 and 3

Similar Partitioning is needed for #2 and #3

  • To be filled out


Proposed timeline for tgt

Proposed Timeline for TGt

  • Metrics/Sub-metrics, Methodology development

Letter

Ballot

Mar’05

Framework

Adoption

Report

In Berlin

Sep’04

Framework

Adoption

Pre-Berlin

9thSep’04

Usage Case #1

Nov’04

Usage Case #2

Usage Case #3

Jan’05


Next steps

Next Steps

  • Get agreement on the framework and approach

  • Get agreement that the three usage cases presented are the one we can focus on

  • Align in Berlin meeting with wider audience in TG

  • Develop the three usage cases further

    • Sanitize and update the metrics and sub-metrics

    • Hone in on the Methodology components, test environment, procedure, etc.

    • Finalizing the test case templates for reporting and analysis

  • Usable outputs for Usages #1, #2, #3 (see timeline)

  • Draft recommended practices specification

    • Include definitions and terminology


Backup

Backup


Parking lot for metrics and sub metrics

Parking Lot for Metrics and Sub-metrics

Proposing a “parking-lot” list of parameters – need to determine whether they are metrics or sub-metrics, or part of the methodology, and which of the usage cases they apply to …

Performance metrics by OSI model layers


  • Login