Skip this Video
Download Presentation
Automated SIP Interoperability Testing

Loading in 2 Seconds...

play fullscreen
1 / 24

Automated SIP Interoperability Testing - PowerPoint PPT Presentation

  • Uploaded on

Automated SIP Interoperability Testing. Archana Rao and Henning Schulzrinne Department of Computer Science Columbia University SIP 2009 (Paris, January 2009). Overview. The problem of SIP interoperability Far from perfect

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Automated SIP Interoperability Testing' - gyala

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Automated SIP Interoperability Testing

Archana Rao and Henning Schulzrinne

Department of Computer Science

Columbia University

SIP 2009 (Paris, January 2009)

  • The problem of SIP interoperability
    • Far from perfect
    • Damages “brand”, harms customers, encourages single-vendor deployments
  • Testbed Overview
    • Architecture
    • Components
  • Experiments
    • Interoperability study
    • End-client characteristics
  • Summary

Is it a real problem ?

Hi Henning,You may remember me from IETFs past -- I haven't attended any in some time because I couldn't find any really interesting projects. I'm finallygetting into SIP. I've got Speakeasy VoIP service, two sipphone accounts, a Cisco 7960 and a copy of x-ten on my Mac. And I still can't make it work. Voice flows in one direction only. I'm not even behind a NAT or firewall -- both machines have global addresses, with no port translations or firewalls.

I've been working with Internet protocols for over 20 years. I've implemented and contributed to them. And if *I* can't figure out how to make this stuff work, how is the average grandmother expected to do so?

SIP is unbelievably complex, with extraordinarily confusing terms. There must be half a dozen different "names" -- Display Name, User Name, Authorization User Name, etc -- and a dozen "proxies". Even the word "domain" is overloaded a half dozen different ways. This is ridiculous!Sorry. I just had to get this off my chest. Regards, Phil Karn.

Excerpts from an email posted on IETF RAI mailing list –


interoperability issues
Interoperability issues

Why do we have interoperability problems?

  • SIP is complex
    • More than 150 RFCs and 500 active Internet Drafts
    • Efforts like – “draft-ieft-sip-hitchhikers-guide” help to a certain extent, but not sufficient
  • SIP has no proposed architecture/profile
    • SIP usage in variety of environments and domains make it impossible
    • Standardization from IETF is different from that of ISO/ITU
  • SIP is continually evolving
    • Non trivial for implementers to follow the life-cycle from IDs to RFCs
  • Walled gardens, non-standard environments
    • Vendors need to make products that work in their customer’s environment
  • Poor implementations, intentional non-interoperability etc.
improving interoperability
Improving interoperability

Current efforts to combat SIP interoperability issues

  • SIP Interoperability (SIPit) events
    • Week long gatherings of SIP implementers to test interoperability
    • Coordinated by SIP Forum
  • SIP Forum
    • SIP Connect 1.0 (and 1.1)  outsourced enterprise VoIP
  • IETF Basic Level of Interoperability for SIP Services (BLISS)
    • Focus on resolving interpretability issues in SIP features
    • Line sharing, call parking, automated call handling and call queuing
  • University of New Hampshire – InterOperability Laboratory
    • Vendor-neutral lab dedicated to testing data networking technologies
    • 20 different testing programs, each costing about $20,000 per year

So, isn’t the problem solved?

  • These forums focus on details of advanced SIP features
  • Basic-level of interoperability between innumerable SIP devices in the market
    • Can I take any SIP phone and make a VoIP call, through any SIP service provider?

How do we go about it?

1. Identify the basic real-world scenarios for SIP registration and session establishment

2. Use our VoIP testbed to configure a variety of SIP infrastructure

3. Study the behavior of SIP end-clients and servers for interoperability

studying interoperability
Studying interoperability
  • Define a minimum set of call flows constituting basic interoperability
    • RFC 3665, RFC 3666, RFC 4317
  • Use a VoIP testbed to realize a variety of real-world SIP infrastructure setups
    • Columbia VoIP testbed
  • Study the behavior of SIP end-clients and servers in each of these scenarios
    • Interoperability matrices
    • Categorization of common issues 
      • Lack of clarity in the specification
      • Implementation of an older specification
      • Incomplete implementation of the specification
      • Incorrect implementation of the specification
      • Failure against robustness tests
columbia voip testbed
Columbia VoIP testbed


  • VoIP infrastructure for experimentation, analysis, testing, prototyping and deployment of SIP/VoIP components in a variety of environments.
  • Part of a multi-university research project supported by NSF
    • University of North Texas, Purdue University and University of California, Davis


  • There’re no telecom testbeds available for research & experimentation
  • VoIP implementations & experiments have lacked scientific rigor
    • Continuously emerging standards
    • Need to develop repeatable, accurate test frameworks
  • Realistic VoIP experiments require a distributed testbed

your equipment here :-)

columbia voip testbed servers
Columbia VoIP testbed: servers
  • 5 SIP servers
    • Asterisk, Pbxnsip, Sipd, SER, OpenSER
  • 3 Platforms
    • Microsoft Windows XP
    • Linux Fedora Core 6
    • Sun Solaris X
  • 3-way Connectivity
    • the Internet
    • VPN
    • PSTN
columbia voip testbed uas
Columbia VoIP testbed: UAs
  • 20+ SIP hard-phones, soft-phones, WI-FI phones with different capabilities
  • More details –
nsf voip testbed








NSF VoIP testbed

Image courtesy:

columbia voip testbed experiments
Columbia VoIP testbed experiments

Planned or ongoing

  • Interoperability study
    • Can any two SIP devices talk to each other?
  • SIP end-client characteristics
    • Signaling and codec features in SIP devices
    • Robustness
    • VoIP with/despite NAT
  • Performance issues
    • Signaling delay, voice mouth-ear delay, packet loss resilience
    • Servers – Scalability, maximum capacity
  • Security
    • Separate testbed for DOS and other attacks

work in progress

interoperability study call flows
Interoperability study: call flows

Registration (RFC 3665)

  • Successful new registration
  • Unsuccessful registration
  • Cancellation of registration
  • Update of contact list

Session Establishment (RFC 3665)

  • Successful session establishment
  • Session establishment through two proxies
  • Session establishment with multiple proxy authentication
  • Successful session with proxy failure
  • Unsuccessful no answer
  • Unsuccessful busy
  • Unsuccessful no response from user agent
  • Unsuccessful temporarily unavailable

Codec Negotiation (RFC 4317)

  • Audio and DTMF
  • Audio, video and DTMF
  • Audio and video codec reordering
interoperability study matrix
Interoperability study: matrix

Interoperability Matrix

Test case #1 - Successful new registration

interoperability study auth name
Interoperability study: auth name

Issue 1 – Use of different formats for authentication name

  • Authorization: Digest username=“user1”,realm=“”,nonce=“xxx”, uri=“”,response=“yyy”, algorithm=MD5

Authorization: Digest username=“sip:[email protected]”, realm=“proxy01.sipphone. com”,nonce=“xxx”,uri=“”,response=“yyy”, algorithm=MD5

  • Implication
    • Registration and call setup failure, if both formats are not supported by the UAs
  • Instances
    • Polycom PVX/VSX, Wengophone don’t register with 3Com / sipd proxies
  • Workaround
    • Use Asterisk between Polycom PVX and 3Com proxy
interoperability study rport
SIP server


INVITE (Dest port:5060)

180 Ringing (Src port:12345)

200 OK (Src port:12345)

ACK (Dest port:12345)

200 OK (Src port:12345)

ACK (Dest port:12345)

Interoperability study: rport

Issue 2 – Behavior in the absence of rport parameter

  • Implication
    • Incorrect/incomplete signaling resulting in multiple retransmissions and failed transaction
  • Instances
    • Calls to Cisco 7940 via Sipphone proxy are always unsuccessful
    • Calls from Polycom VSX via 3com or sipd proxies are always unsuccessful
  • Workaround
    • Nothing
interoperability study codecs
Interoperability Study: codecs

Issue 3 – Codec Negotiation

  • Implication – Audio/Video failure (mostly unidirectional, but bidirectional sometimes)
  • Problem Instances
    • Audio failure when Cisco calls voicemail service of sipphone provider
    • Video failure when Polycom PVX calls Xlite via Asterisk
  • [Offer]


o=Cisco-SIPUA 28422 0 IN IP4

s= SIP Call

t=0 0

m=audio 31030 RTP/AVP 0 8 18 101

c=IN IP4

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:18 G729/0

a=fmtp:18 annexb=no

a=rtpmap:101 telephone-event/8000

a=fmtp:101 0-15


  • [Answer]


o=Cisco-SIPUA 28422 0 IN IP4

s= SIP Call

t=0 0

m=audio 31030 RTP/AVP 18 0 8 101

c=IN IP4

a=rtpmap:18 G729/0

a=fmtp:18 annexb=no

a=rtpmap:0 PCMU/8000

a=rtpmap:8 PCMA/8000

a=rtpmap:101 telephone-event/8000

a=fmtp:101 0-15


ua characteristics features
UA characteristics: features
  • Basic
    • Configuration, number of lines, call hold, call forwarding, DND, NAT support
  • Advanced
    • Encryption, symmetric RTP, conferencing, audio/video codecs, PoE
    • Audio quality – silence suppression, echo cancellation, comfort noise
    • Transport, signaling and support protocols – TCP, TLS, HTTP, DHCP, DNS, TFTP, NTP
  • More details at –
ua characteristics robustness
UA characteristics: robustness


  • SIP Torture Tests (RFC4475)
    • A set of SIP messages aimed at stressing the SIP parser, based on experiences at SIPit.
    • Test suite with 4500+ INVITE requests, developed by University of Oulu

Example Results

ua characteristics nats
UA characteristics: NATs


  • Problematic for both signaling (SIP) and media transfer (RTP)
  • Circumventing NAT effect – solutions exist for both end-clients and proxies
  • Quest to overcome this, could lead to techniques uglier than NAT itself!

NAT test setup

  • NAT realization - Linux iptables, CLICK router
  • Studying the behavior in different scenarios
    • NAT support enabled in both the proxy server and end-client
    • NAT support enabled in only one of them etc
  • Aiming to analyze the existing practices – the good, the bad and the ugly!
ua with nats
UA with NATs

SIP devices in NAT environment

what now
What now?


  • Tests indicate that basic-level interoperability is far from perfect
  • Instances of interoperability failures illustrated in our IETF workshop paper
  • Achieving Interoperability shouldn’t be left just to vendors
    • non-interoperability damages SIP brand, causes grief for third parties and harms customers

Going ahead – some ideas

  • Establish designated interop liaison for each vendor
    • calling customer support unlikely to be useful
  • Encourage vendors to publish interoperability reports
    • in standard format
  • Provide self-certification tests, supported by a remotely accessible test rig

Current status

  • The VoIP testbed & VPN connectivity is operational
  • Systematic study of real-world interoperability issues
    • Basic-level interoperability is nowhere near 100%
    • IETF 70 – The 1st SIP Forum Workshop on SIP Interoperability
  • Evaluating SIP end client characteristics

Testbed plans

  • Experiments on NAT, performance, security
  • Make the testbed accessible and useful for engineers & deployments
  • Create a knowledge repository about SIP devices & tools for interoperability testing