Skip this Video
Download Presentation
RDDP Is Coming Soon

Loading in 2 Seconds...

play fullscreen
1 / 17

RDDP Is Coming Soon - PowerPoint PPT Presentation

  • Uploaded on

Remote Direct Memory Access (RDMA) over IP PFLDNet 2003, Geneva Stephen Bailey, Sandburst Corp., [email protected] Allyn Romanow, Cisco Systems, [email protected] RDDP Is Coming Soon. “ST [RDMA] Is The Wave Of The Future” – S Bailey & C Good, CERN 1999 Need: standard protocols host software

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about ' RDDP Is Coming Soon' - weston

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Remote Direct Memory Access (RDMA) over IPPFLDNet 2003, GenevaStephen Bailey, Sandburst Corp., [email protected] Romanow, Cisco Systems, [email protected]

rddp is coming soon
RDDP Is Coming Soon

“ST [RDMA] Is The Wave Of The Future” – S Bailey & C Good, CERN 1999

  • Need:
    • standard protocols
    • host software
    • accelerated NICs (RNICs)
    • faster host buses (for > 1G)
  • Vendors are finally serious:

Broadcom, Intel, Agilent, Adaptec, Emulex, Microsoft, IBM, HP (Compaq, Tandem, DEC), Sun, EMC, NetApp, Oracle, Cisco & many, many others

  • Motivation
  • Architecture
  • Open Issues
cfp sigcomm workshop
CFP SigComm Workshop
  • NICELI SigComm 03 Workshop

Workshop on Network-I/O Convergence: Experience, Lessons, Implications

high speed data transfer
High Speed Data Transfer
  • Bottlenecks
    • Protocol performance
    • Router performance
    • End station performance, host processing
      • CPU Utilization
      • The I/O Bottleneck
        • Interrupts
        • TCP checksum
        • Copies
what is rdma
What is RDMA?
  • Avoids copying by allowing network adapter under control of application to steer data directly into application buffers
  • Bulk data transfer or kernel bypass for small messages
  • Grid, cluster, supercomputing, data centers
  • Historically, special purpose fabrics – Fibre Channel, VIA, Infiniband, Quadrics, Servernet
traditional data center






(Fibre Channel)

A Machine






Traditional Data Center

The World


why rdma over ip business case
Why RDMA over IP? Business Case
  • TCP/IP not used for high bandwidth interconnection, host processing costs too high
  • High bandwidth transfer to become more prevalent – 10 GE, data centers
  • Special purpose interfaces are expensive
  • IP NICs are cheap, volume
the technical problem i o bottleneck
The Technical Problem- I/O Bottleneck
  • With TCP/IP host processing can’t keep up with link bandwidth, on receive
  • Per byte costs dominate, Clark (89)
  • Well researched by distributed systems community, mid 1990’s. Industry experience.
  • Memory bandwidth doesn’t scale, processor memory performance gap– Hennessy(97), D.Patterson, T. Anderson(97),
  • Stream benchmark

Using IP transports (TCP & SCTP) requires data copying








User Buffer

Data copies

why is copying important
Why Is Copying Important?
  • Heavy resource consumption @ high speed (1Gbits/s and up)
    • Uses large % of available CPU
    • Uses large fraction of avail. bus bw – min 3 trips across the bus

64 KB window, 64 KB I/Os, 2P 600 MHz PIII, 9000 B MTU

what s in rdma for us
What’s In RDMA For Us?

Network I/O becomes `free’ (still have latency though)

1750 machines using 0% CPU for I/O

2500 machines using 30% CPU for I/O

approaches to copy reduction
Approaches to Copy Reduction
  • On-host – Special purpose software and/or hardware e.g., Zero Copy TCP, page flipping
    • Unreliable, idiosyncratic, expensive
  • Memory to memory copies, using network protocols to carry placement information
    • Satisfactory experience – Fibre Channel, VIA, Servernet
  • FOR HARDWARE, not software
rdma over ip standardization
RDMA over IP Standardization
  • IETF RDDP Remote Direct Data Placement WG
  • RDMAC RDMA Consortium
rdma over ip architecture


RDMA control




RDMA over IP Architecture

Two layers:

  • DDP – Direct Data Placement
  • RDMA - control
upper and lower layers
Upper and Lower Layers
  • ULPs- SDP Sockets Direct Protocol, iSCSI, MPI
  • DAFS is standardized NFSv4 on RDMA
  • SDP provides SOCK_STREAM API
  • Over reliable transport – TCP, SCTP
open issues
Open Issues
  • Security
  • TCP order processing, framing
  • Atomic ops
  • Ordering constraints – performance vs. predictability
  • Other transports, SCTP, TCP, unreliable
  • Impact on network & protocol behaviors
  • Next performance bottleneck?
  • What new applications?
  • Eliminates the need for large MTU (jumbos)?