realizing the performance potential of the virtual interface architecture n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Realizing the Performance Potential of the Virtual Interface Architecture PowerPoint Presentation
Download Presentation
Realizing the Performance Potential of the Virtual Interface Architecture

Loading in 2 Seconds...

play fullscreen
1 / 26

Realizing the Performance Potential of the Virtual Interface Architecture - PowerPoint PPT Presentation


  • 111 Views
  • Uploaded on

Realizing the Performance Potential of the Virtual Interface Architecture. Evan Speight, Hazim Abdel-Shafi, and John K. Bennett Rice University, Dep. Of Electrical and Computer Engineering Presented by Constantin Serban, R.U. VIA Goals.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Realizing the Performance Potential of the Virtual Interface Architecture' - tocho


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
realizing the performance potential of the virtual interface architecture

Realizing the Performance Potential of the Virtual Interface Architecture

Evan Speight, Hazim Abdel-Shafi, and John K. Bennett

Rice University, Dep. Of Electrical and Computer Engineering

Presented by Constantin Serban, R.U.

via goals
VIA Goals
  • Communication infrastructure for System Area Networks (SANs)
  • Targets mainly high speed cluster applications
  • Efficiently harnesses the communication performance of underlying networks
trends
Trends
  • The peak bandwidth increase two order of magnitude over past decade while user latency decreased modestly.
  • The latency introduced by the protocol is typically several times the latency of the transport layer.
  • The problem becomes acute especially for small messages
targets
Targets

VI architecture addresses the following issues:

  • Decrease the latency especially for small messages (used in synchronization)
  • Increase the aggregate bandwidth (only a fraction of the peak bandwidth is utilized)
  • Reduce the CPU processing due to the message overhead
overhead
Overhead

Overhead mainly comes from two sources:

  • Every network access requires one-two traps into the kernel
    • user/kernel mode switch is time consuming
  • Usually two data copies occur:
    • From the user buffer to the message passing API
    • From message layer to the kernel buffer
via approach
VIA approach
  • Remove the kernel from the critical path
    • Moving communication code out of the kernel into user space
  • Provide 0-copy protocol
    • Data is sent/received directly into the user buffer, no message copy is performed
slide7
VIA emerged as a standardization effort from Compaq, Intel, and Microsoft

It was built on several academic ideas:

  • The main architecture most similar to U-Net
  • Essential features derived from VMMC

Among current implementations :

    • GigaNet cLan – VIA implemented in hardware
    • Tandem ServerNet –VIA software driver emulated
    • Myricom Myrinet - software emulated in firmware
via operations
VIA operations

Set-Up/Tear-Down :

  • VIA is point-to-point connection oriented protocol
  • VI-endpoint : the core concept in VIA
  • Register/De-Register Memory
  • Connect/Disconnect
  • Transmit
  • Receive
  • RDMA
via operations1
VIA operations

Set-Up/Tear-Down :VIA is point-to-point connection oriented protocol

  • VI-endpoint : the core concept in VIA
  • VipCreateVi function creates a VI endpoint in the user space.
  • The user-level library passes the call to the kernel agent which passes the creation information to the NIC.
  • OS thus controls the application access to the NIC
via operations cont d
VIA operations - cont’d

Register/De-Register Memory:

  • All data buffers and descriptors reside in a registered memory
  • NIC performs DMA I/O operation in this registered memory
  • Registration pins down the pages into the physical memory and provides a handle to manipulate the pages and transfer the addresses to the NIC
  • It is performed once, usually at the beginning of the communication session
via operations cont d1
VIA operations - cont’d

Connect/Disconnect:

  • Before communication, each endpoint is connected to a remote endpoint
  • The connection is passed to the kernel agent and down to the NIC
  • VIA does not define any addressing scheme, existing schemes can be used in various implementations
via operations cont d2
VIA operations - cont’d

Transmit/receive:

  • The sender builds a descriptor for the message to be sent. The descriptor points to the actual data buffer. Both descriptor and data buffer resides in a registered memory area.
  • The application then posts a doorbell to signal the availability of the descriptor.The doorbell contains the address of the descriptor.
  • The doorbells are maintained in an internal queue inside the NIC
via operations cont d3
VIA operations - cont’d

Transmit/receive (cont’d):

  • Meanwhile, the receiver creates a descriptor that points to an empty data buffer and posts a doorbell in the receiver NIC queue
  • When the doorbell in the sender queue has reached the top of the queue, through a double indirection the data is sent into the network.
  • The first doorbell/ descriptor is picked up from the receiver queue and the buffer is filled out with data
via operations cont d4
VIA operations - cont’d

RDMA:

  • As a mechanism derived from VMMC, VIA allows Remote DMA operations:

RDMA Read and Write

  • Each node allocates a receive buffer and registers it with the NIC. Additional structures that contain read and write pointers to the receive buffers are exchanged during connection setu
  • Each node can read and write to the remote node address directly.
  • These operations posts potential implementation problems.
evaluation benchmarks
Evaluation Benchmarks
  • Two VI implementations :
    • GigaNet cLan B:125MB/sec, Latency 480ns
    • Tandem ServerNet, 50MB/S, Latency 300ns
  • Performance measured:
    • Bandwidth and Latency
    • Poling vs. Blocking
    • CPU Utilization
mpi performance using via
MPI performance using VIA
  • The challenge is to deliver performance to distributed application
  • Software layers such MPI are mostly used between VIA and the application: provide increased usability but they bring additional overhead
  • How to optimize this layer in order to use it efficiently with VIA ?
mpi observations
MPI observations
  • Difference between MPI-UDP and MPI-VIA-baseline is remarkable
  • MPI-VIA-baseline is dramatically far from VIA-Native
  • Several improvements proposed to shift MPI-Via to be closer to VIA native : reduce MPI overhead
mpi improvements
MPI Improvements
  • Eliminating unnecessary copies:

MPI UDP and VIA use a single set of receiving buffers, thus data should be copied to the application : allow the user to register any buffer

  • Choosing a synchronization primitive:

All synchronization formerly using OS constructs/events. Better implementation using swap processor commands

  • No Acknowledge:

Remove the acknowledge of the message by switching to a reliable VIA mode

via disadvantages
VIA - Disadvantages
  • Polling vs. blocking synchronization – a tradeoff between CPU consumption and overhead
  • Memory registration: locking large amount of memory makes virtual memory mechanisms inefficient. Registering / deregistering on the fly is slow
  • Point-to-point vs. multicast: VIA lacks multicast primitives. Implementing multicast over the actual mechanism, makes communication inefficient
conclusion
Conclusion
  • Small latency for small messages. Small messages have a strong impact on application behavior
  • Significant improvement over UDP communication (still after recent TCP/UDP hardware implementations?)
  • At the expense of an uncomfortable API