Automated SIP Interoperability Testing
1 / 24

Automated SIP Interoperability Testing - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Automated SIP Interoperability Testing. Archana Rao and Henning Schulzrinne Department of Computer Science Columbia University SIP 2009 (Paris, January 2009). Overview. The problem of SIP interoperability Far from perfect

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Automated SIP Interoperability Testing

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Automated SIP Interoperability Testing

Archana Rao and Henning Schulzrinne

Department of Computer Science

Columbia University

SIP 2009 (Paris, January 2009)


  • The problem of SIP interoperability

    • Far from perfect

    • Damages “brand”, harms customers, encourages single-vendor deployments

  • Testbed Overview

    • Architecture

    • Components

  • Experiments

    • Interoperability study

    • End-client characteristics

  • Summary


Is it a real problem ?

Hi Henning,You may remember me from IETFs past -- I haven't attended any in some time because I couldn't find any really interesting projects. I'm finallygetting into SIP. I've got Speakeasy VoIP service, two sipphone accounts, a Cisco 7960 and a copy of x-ten on my Mac. And I still can't make it work. Voice flows in one direction only. I'm not even behind a NAT or firewall -- both machines have global addresses, with no port translations or firewalls.

I've been working with Internet protocols for over 20 years. I've implemented and contributed to them. And if *I* can't figure out how to make this stuff work, how is the average grandmother expected to do so?

SIP is unbelievably complex, with extraordinarily confusing terms. There must be half a dozen different "names" -- Display Name, User Name, Authorization User Name, etc -- and a dozen "proxies". Even the word "domain" is overloaded a half dozen different ways. This is ridiculous!Sorry. I just had to get this off my chest. Regards, Phil Karn.

Excerpts from an email posted on IETF RAI mailing list –


Interoperability issues

Why do we have interoperability problems?

  • SIP is complex

    • More than 150 RFCs and 500 active Internet Drafts

    • Efforts like – “draft-ieft-sip-hitchhikers-guide” help to a certain extent, but not sufficient

  • SIP has no proposed architecture/profile

    • SIP usage in variety of environments and domains make it impossible

    • Standardization from IETF is different from that of ISO/ITU

  • SIP is continually evolving

    • Non trivial for implementers to follow the life-cycle from IDs to RFCs

  • Walled gardens, non-standard environments

    • Vendors need to make products that work in their customer’s environment

  • Poor implementations, intentional non-interoperability etc.

Improving interoperability

Current efforts to combat SIP interoperability issues

  • SIP Interoperability (SIPit) events

    • Week long gatherings of SIP implementers to test interoperability

    • Coordinated by SIP Forum

  • SIP Forum

    • SIP Connect 1.0 (and 1.1)  outsourced enterprise VoIP

  • IETF Basic Level of Interoperability for SIP Services (BLISS)

    • Focus on resolving interpretability issues in SIP features

    • Line sharing, call parking, automated call handling and call queuing

  • University of New Hampshire – InterOperability Laboratory

    • Vendor-neutral lab dedicated to testing data networking technologies

    • 20 different testing programs, each costing about $20,000 per year


So, isn’t the problem solved?

  • These forums focus on details of advanced SIP features

  • Basic-level of interoperability between innumerable SIP devices in the market

    • Can I take any SIP phone and make a VoIP call, through any SIP service provider?

How do we go about it?

1. Identify the basic real-world scenarios for SIP registration and session establishment

2. Use our VoIP testbed to configure a variety of SIP infrastructure

3. Study the behavior of SIP end-clients and servers for interoperability

Studying interoperability

  • Define a minimum set of call flows constituting basic interoperability

    • RFC 3665, RFC 3666, RFC 4317

  • Use a VoIP testbed to realize a variety of real-world SIP infrastructure setups

    • Columbia VoIP testbed

  • Study the behavior of SIP end-clients and servers in each of these scenarios

    • Interoperability matrices

    • Categorization of common issues 

      • Lack of clarity in the specification

      • Implementation of an older specification

      • Incomplete implementation of the specification

      • Incorrect implementation of the specification

      • Failure against robustness tests

Columbia VoIP testbed


  • VoIP infrastructure for experimentation, analysis, testing, prototyping and deployment of SIP/VoIP components in a variety of environments.

  • Part of a multi-university research project supported by NSF

    • University of North Texas, Purdue University and University of California, Davis


  • There’re no telecom testbeds available for research & experimentation

  • VoIP implementations & experiments have lacked scientific rigor

    • Continuously emerging standards

    • Need to develop repeatable, accurate test frameworks

  • Realistic VoIP experiments require a distributed testbed

your equipment here :-)

Columbia VoIP testbed: servers

  • 5 SIP servers

    • Asterisk, Pbxnsip, Sipd, SER, OpenSER

  • 3 Platforms

    • Microsoft Windows XP

    • Linux Fedora Core 6

    • Sun Solaris X

  • 3-way Connectivity

    • the Internet

    • VPN

    • PSTN

Columbia VoIP testbed: UAs

  • 20+ SIP hard-phones, soft-phones, WI-FI phones with different capabilities

  • More details –









NSF VoIP testbed

Image courtesy:

Columbia VoIP testbed experiments

Planned or ongoing

  • Interoperability study

    • Can any two SIP devices talk to each other?

  • SIP end-client characteristics

    • Signaling and codec features in SIP devices

    • Robustness

    • VoIP with/despite NAT

  • Performance issues

    • Signaling delay, voice mouth-ear delay, packet loss resilience

    • Servers – Scalability, maximum capacity

  • Security

    • Separate testbed for DOS and other attacks

work in progress

Interoperability study: call flows

Registration (RFC 3665)

  • Successful new registration

  • Unsuccessful registration

  • Cancellation of registration

  • Update of contact list

Session Establishment (RFC 3665)

  • Successful session establishment

  • Session establishment through two proxies

  • Session establishment with multiple proxy authentication

  • Successful session with proxy failure

  • Unsuccessful no answer

  • Unsuccessful busy

  • Unsuccessful no response from user agent

  • Unsuccessful temporarily unavailable

Codec Negotiation (RFC 4317)

  • Audio and DTMF

  • Audio, video and DTMF

  • Audio and video codec reordering

Interoperability study: matrix

Interoperability Matrix

Test case #1 - Successful new registration

Interoperability study: auth name

Issue 1 – Use of different formats for authentication name

  • Authorization: Digest username=“user1”,realm=“”,nonce=“xxx”, uri=“”,response=“yyy”, algorithm=MD5

    Authorization: Digest username=“”, realm=“proxy01.sipphone. com”,nonce=“xxx”,uri=“”,response=“yyy”, algorithm=MD5

  • Implication

    • Registration and call setup failure, if both formats are not supported by the UAs

  • Instances

    • Polycom PVX/VSX, Wengophone don’t register with 3Com / sipd proxies

  • Workaround

    • Use Asterisk between Polycom PVX and 3Com proxy

SIP server


INVITE (Dest port:5060)

180 Ringing (Src port:12345)

200 OK (Src port:12345)

ACK (Dest port:12345)

200 OK (Src port:12345)

ACK (Dest port:12345)

Interoperability study: rport

Issue 2 – Behavior in the absence of rport parameter

  • Implication

    • Incorrect/incomplete signaling resulting in multiple retransmissions and failed transaction

  • Instances

    • Calls to Cisco 7940 via Sipphone proxy are always unsuccessful

    • Calls from Polycom VSX via 3com or sipd proxies are always unsuccessful

  • Workaround

    • Nothing

Interoperability Study: codecs

Issue 3 – Codec Negotiation

  • Implication – Audio/Video failure (mostly unidirectional, but bidirectional sometimes)

  • Problem Instances

    • Audio failure when Cisco calls voicemail service of sipphone provider

    • Video failure when Polycom PVX calls Xlite via Asterisk

  • [Offer]


    o=Cisco-SIPUA 28422 0 IN IP4

    s= SIP Call

    t=0 0

    m=audio 31030 RTP/AVP 0 8 18 101

    c=IN IP4

    a=rtpmap:0 PCMU/8000

    a=rtpmap:8 PCMA/8000

    a=rtpmap:18 G729/0

    a=fmtp:18 annexb=no

    a=rtpmap:101 telephone-event/8000

    a=fmtp:101 0-15


  • [Answer]


    o=Cisco-SIPUA 28422 0 IN IP4

    s= SIP Call

    t=0 0

    m=audio 31030 RTP/AVP 18 0 8 101

    c=IN IP4

    a=rtpmap:18 G729/0

    a=fmtp:18 annexb=no

    a=rtpmap:0 PCMU/8000

    a=rtpmap:8 PCMA/8000

    a=rtpmap:101 telephone-event/8000

    a=fmtp:101 0-15


UA characteristics: features

  • Basic

    • Configuration, number of lines, call hold, call forwarding, DND, NAT support

  • Advanced

    • Encryption, symmetric RTP, conferencing, audio/video codecs, PoE

    • Audio quality – silence suppression, echo cancellation, comfort noise

    • Transport, signaling and support protocols – TCP, TLS, HTTP, DHCP, DNS, TFTP, NTP

  • More details at –

UA characteristics: robustness


  • SIP Torture Tests (RFC4475)

    • A set of SIP messages aimed at stressing the SIP parser, based on experiences at SIPit.


    • Test suite with 4500+ INVITE requests, developed by University of Oulu

Example Results

UA characteristics: NATs


  • Problematic for both signaling (SIP) and media transfer (RTP)

  • Circumventing NAT effect – solutions exist for both end-clients and proxies

  • Quest to overcome this, could lead to techniques uglier than NAT itself!

NAT test setup

  • NAT realization - Linux iptables, CLICK router

  • Studying the behavior in different scenarios

    • NAT support enabled in both the proxy server and end-client

    • NAT support enabled in only one of them etc

  • Aiming to analyze the existing practices – the good, the bad and the ugly!

UA with NATs

SIP devices in NAT environment

What now?


  • Tests indicate that basic-level interoperability is far from perfect

  • Instances of interoperability failures illustrated in our IETF workshop paper

  • Achieving Interoperability shouldn’t be left just to vendors

    • non-interoperability damages SIP brand, causes grief for third parties and harms customers

Going ahead – some ideas

  • Establish designated interop liaison for each vendor

    • calling customer support unlikely to be useful

  • Encourage vendors to publish interoperability reports

    • in standard format

  • Provide self-certification tests, supported by a remotely accessible test rig


Current status

  • The VoIP testbed & VPN connectivity is operational

  • Systematic study of real-world interoperability issues

    • Basic-level interoperability is nowhere near 100%

    • IETF 70 – The 1st SIP Forum Workshop on SIP Interoperability

  • Evaluating SIP end client characteristics

Testbed plans

  • Experiments on NAT, performance, security

  • Make the testbed accessible and useful for engineers & deployments

  • Create a knowledge repository about SIP devices & tools for interoperability testing

  • Login