1 / 17

Reliable Datagram Sockets for High-Level Performance and Scalability

This project aims to provide a reliable datagram service with high performance, scalability, and availability, while simplifying application code. Maintaining the sockets API and ensuring application code portability are key objectives. The design involves RDS registering with the kernel as a driver, creating RDS sockets, and implementing connection models that are connectionless and transparent to applications. The high availability feature allows for failover using RC and on-demand connection setup. Preliminary performance data shows promising results on OpenIB setups. The project status includes functional completion of Z-copy and stability testing with Netperf and Oracle unit test.

shaunt
Download Presentation

Reliable Datagram Sockets for High-Level Performance and Scalability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Reliable Datagram Sockets(RDS) Ranjit Pandit SilverStorm Technologies rpandit@silverstorm.com

  2. Agenda • Goals • High Level Design • Current status • Preliminary performance data • Future work

  3. Goals • Provide reliable datagram service • performance • scalability • high availability • simplify application code • Maintain sockets API • application code portability • faster time-to-market Keep It Simple !!!

  4. Stack Overview UDP Applications Oracle 10g Socket Applications User Kernel TCP UDP SDP RDS IP IPoIB Openib Access Layer Host Channel Adapter

  5. High Level Design • RDS registers with the kernel as driver for Address Family PF_INET_OFFLOAD and Type SOCK_DGRAM • Application creates a RDS socket with socket(2) • arg1 = PF = PF_INET_OFFLOAD • arg 2 = Type = SOCK_DGRAM • socket(2) API supported • socket, bind, ioctl, sendmsg, recvmsg, poll, getsockopt/setsockopt

  6. Connection model • Application connectionless • Rds maintains node-to-node connection • IP addressing • Uses CMA • on-demand connection setup • connect on first sendmsg()or data recv • disconnect on error or policy like inactivity • Connection setup/teardown transparent to applications Applicationconnectionless

  7. Data and Control Channel • Uses RC QP for node level connections • Data and Control QPs per session • Selectable MTU • b-copy send/recv • h/w flow control

  8. sn s2 s1 S1 recvmsg() RC QP RC QP P2 Pn User P1 P1 … sendmsg(node2) Rds Rds Kernel Node 1 Node 2

  9. Send • Connection established on first send • sendmsg() • allows send pipelining • ENOBUF returned if insufficient send buffers, application retries

  10. Receive • Identical to UDP recvmsg() • similar blocking/non-blocking behavior • “Slow” receiver ports are stalled at sender side • combination of activity (LRU) and memory utilization used to detect slow receivers • sendmsg() to stalled destination port returns EWOULDBLOCK, application can retry • Blocking socket can wait for unblock • recvmsg() on a stalled port un-stalls it

  11. High Availability (failover) • Use of RC and on-demand connection setup allows HA • connection setup/teardown transparent to applications • every sendmsg() could “potentially” result in a connection setup • if a path fails, connection is torn down, next send can connect on an alternate path (different port or different HCA)

  12. Preliminary performance Rds on Openib *Dual 2.4GHz Xeon 2G memory 4x PCI-X HCA **Sdp ~3700Mb/sec TCP_STREAM

  13. Preliminary performance Rds on OpenIB *Dual 2.4GHz Xeon 2G memory 4x PCI-X HCA **Sdp ~3700Mb/sec TCP_STREAM

  14. Preliminary performance Rds on OpenIB

  15. Status in OpenIB • Z-copy • Functionally 98% complete • Running Netperf • Running Oracle unit test (crload) stable today • Code checked into contrib/silverstorm/ https://openib.org/svn/trunk/contrib/silverstorm/rds/

  16. Future • AIO • Z-copy • Shared recv queue

  17. Q&A

More Related