Quality and performance evaluation of voip end points
1 / 28

- PowerPoint PPT Presentation

  • Updated On :

Quality and Performance Evaluation of VoIP End-points. Wenyu Jiang Henning Schulzrinne Columbia University NYMAN 2002 September 3, 2002. Motivations. The quality of VoIP depends on both the network and the end-points Extensive QoS literature on network performance, e.g., IntServ, DiffServ

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about '' - phyre

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Quality and performance evaluation of voip end points l.jpg

Quality and Performance Evaluation of VoIP End-points

Wenyu Jiang

Henning Schulzrinne

Columbia University

NYMAN 2002

September 3, 2002

Motivations l.jpg

  • The quality of VoIP depends on both the network and the end-points

  • Extensive QoS literature on network performance, e.g., IntServ, DiffServ

    • Focus is on limiting network loss & delay

  • Little is known about the behavior of VoIP end-points

Performance metrics for voip end points l.jpg
Performance Metrics for VoIP End-points

  • Mouth-to-ear (M2E) delay

    • C.f. network delay

  • Clock skew

    • Whether it causes any voice glitches

    • Amount of clock drift

  • Silence suppression behavior

    • Whether the voice is clipped (depends much on hangover time)

    • Robustness to non-speech input, e.g., music

  • Robustness to packet loss

    • Voice quality under packet loss

Measurement approach l.jpg
Measurement Approach

  • Capture both original and output audio

  • Use “adelay” program to measure M2E delay

  • Assume a LAN environment by default

    • Serve as a baseline of reference, or lower bound

Voip end points tested l.jpg
VoIP End-points Tested

  • Hardware End-points

    • Cisco, 3Com and Pingtel IP phones

    • Mediatrix 1-line SIP/PSTN Gateway

  • Software clients

    • Microsoft Messenger, NetMeeting (Win2K, WinXP)

    • Net2Phone (NT, Win2K, Win98)

    • Sipc/RAT (Solaris, Ultra-10)

      • Robust Audio Tool (RAT) from UCL as media client

  • Operating parameters:

    • In most cases, codec is G.711 -law, packet interval is 20ms

Example m2e delay plot l.jpg
Example M2E Delay Plot

  • 3Com to Cisco, shown with gaps > 1sec

  • Delay adjustments correlate with gaps, despite 3Com phone has no silence suppression

Visual illustration of m2e delay drop snapshot 1 l.jpg
Visual Illustration of M2E Delay Drop, Snapshot #1

  • 3Com to Cisco 1-1 case

  • Left/upper channel is original audio

  • Highlighted section shows M2E delay (59ms)

Snapshot 2 l.jpg
Snapshot #2

  • M2E delay drops to 49ms, at time of 4:16

Snapshot 3 l.jpg
Snapshot #3

  • Presence of a gap during the delay change

Effect of rtp m bits on delay adjustments l.jpg
Effect of RTP M-bits on Delay Adjustments

  • Cisco phone sends M-bits, whereas Pingtel phone does not

    • Presence of M-bits results in more adjustments

Sender characteristics l.jpg
Sender Characteristics

  • Certain senders may introduce delay spikes, despite operating on a LAN

Average m2e delays for ip phones and sipc l.jpg
Average M2E Delays for IP phones and sipc

  • Averaging the M2E delay allows more compact presentation of end-point behaviors

  • Receiver (especially RAT) plays an important role in M2E delay

Average m2e delays for pc software clients l.jpg
Average M2E Delays for PC Software Clients

  • Messenger 2000 wins the day

    • Its delay as receiver (68ms) is even lower than Messenger XP, on the same hardware

    • It also results in slightly lower delay as sender

  • NetMeeting is a lot worse (> 400ms)

  • Messenger’s delay performance is similar to or better than a GSM mobile phone.

Delay behaviors for pc clients contd l.jpg
Delay Behaviors for PC Clients, contd.

  • Net2Phone’s delay is also high

    • ~200-500ms

    • V1.5 reduces PC->PSTN delay

    • PC-to-PC calls have fairly high delays

Effect of clock skew cisco to 3com experiment 1 1 l.jpg
Effect of Clock Skew: Cisco to 3Com, Experiment 1-1

  • Symptom of playout buffer underflow

  • Waveforms are dropped

  • Occurred at point of delay adjustment

  • Bugs in software?

Clock skew rates l.jpg
Clock Skew Rates

  • Mostly symmetric between two devices

  • RAT (Ultra-10) has unusually high drift rates, > 300 ppm (parts per million)

    • High clock skews confirmed in many (but not all) PCs and workstations

Drift rates for pc clients l.jpg
Drift Rates for PC Clients

  • Drift Rates not always symmetric!

    • But appears to be consistent between Messenger 2K/XP and Net2Phone on the same PC

    • Existence of 2 clocking circuits in sound card?

Packet loss concealment l.jpg
Packet Loss Concealment

  • Common PLC methods

    • Silence substitution (worst)

    • Packet repetition, with optional fading

    • Extrapolation (one-sided)

    • Interpolation (two-sided), best quality

  • Use deterministic bursty loss pattern

    • 3/100 means 3 consecutive losses out of every 100 packets

    • Easier to locate packet losses

    • Tested 1/100, 3/100, 1/20, 5/100, etc.

Plc behaviors l.jpg
PLC Behaviors

  • Loss tolerance (at 20ms interval)

    • By measuring loss-induced gaps in output audio

    • 3Com and Pingtel phones: 2 packet losses

    • Cisco phone: 3 packet losses

  • Level of audio distortion by packet loss

    • Inaudible at 1/100 for all 3 phones

    • Inaudible at 3/100 and 1/20 for Cisco phone, yet audible to very audible for the other two.

  • Cisco phone is the most robust

    • Probably uses interpolation

Effect of plc on delay l.jpg
Effect of PLC on Delay

  • No affirmative effect on M2E delay

    • E.g., sipc to Pingtel

Silence suppression l.jpg
Silence Suppression

  • Why?

    • Saves bandwidth

    • May reduce processing power (e.g., in conferencing mixer)

    • Facilitates per-talkspurt delay adjustment

  • Key parameters

    • Silence detection threshold

    • Hangover time, to delay silence suppression and avoid end clipping of speech

      • Usually 200ms is long enough [Brady ’68]

Hangover time l.jpg
Hangover Time

  • Measured by feeding ON-OFF waveforms and monitor RTP packets

  • Cisco phone’s is the longest (2.3-2.36 sec), then Messenger (1.06-1.08 sec), then NetMeeting (0.56-0.58 sec)

  • A long hangover time is not necessarily bad, as it reduces voice clipping

    • Indeed, no unnatural gaps are found

    • Does waste a bit more bandwidth

Robustness of silence detectors to music l.jpg
Robustness of Silence Detectors to Music

  • On-hold music is often used in customer support centers

    • Need to ensure music is played without any interruption due to silence suppression

  • Tested with a 2.5-min long soundtrack

  • Messenger starts to generate many unwanted gaps at input level of -24dB

  • Cisco phone is more robust, but still fails from input level of -41.4dB

Acoustic echo cancellation l.jpg
Acoustic Echo Cancellation

  • Important for hands-free/conferencing (business) applications

  • Primary metric: Echo Return Loss (ERL)

    • Measured by LAN-sniffing RTP packets

  • Most IP phones support AEC

    • ERL depends slightly on input level and speaker-phone volume

    • Usually > 40 dB (good AEC performance)

M2e delay under jitter l.jpg
M2E Delay under Jitter

  • Delay properties under the LAN environment serves as a baseline of reference

  • When operating over the Internet:

    • Fixed portion of delay adds to M2E delay as a constant

    • Variable portion (jitter) has a more complex effect

  • Initial test

    • Used typical cable modem delay traces

    • Tested RAT to Cisco

    • No audible distortion due to late loss

    • Added delay is normal

M2e delay under jitter contd l.jpg
M2E Delay under Jitter, contd.

  • Cisco phone generally within expectation

    • Can follow network delay change timely

      • Takes longer (10-20sec) to adapt to decreasing delay

    • Does not overshoot playout delay

  • More end-points to be examined

Artificial Trace

Real Trace with Spikes

Conclusions l.jpg

  • Average M2E Adelay:

    • Low (mostly < 80ms) for hardware IP phones

    • Software clients: lowest for Messenger 2000 (68.5ms)

    • Application (receiver) most vital in determining delay

      • Poor implementation easily undoes good network QoS

  • Clock skew high on SW clients (RAT, Net2Phone)

  • Packet loss concealment quality

    • Acceptable in all 3 IP phones tested, w. Cisco more robust

  • Silence detector behavior

    • Long hangover time, works well for speech input

    • But may falsely predict music as silence

  • Acoustic Echo Cancellation: good on most IP phones

  • Playout delay behavior: good based on initial tests

Future work l.jpg
Future Work

  • Further tests with more end-points on how jitter influences M2E delay

  • Measure the sensitivity (threshold) of various silence detectors

  • Investigate the non-symmetric clock drift phenomena

  • Additional experiments as more brands of VoIP end-points become available