Topics

1. Topics • Overview • Target platform characteristics • Application domain characteristics • Requirements for DQI/NAL • Existing implementations versus own implementation • Global design • Network Abstraction Layer • Descriptor Queue Interface • Possible improvements / future work Januari 29th, 1999 University of Amsterdam slide 1

2. Overview What is DQI/NAL? • A high-performance layer for inter-process communication in a clustered environment What does DQI/NAL stand for? • DQI = Descriptor Queue Interface • NAL = Network Abstraction Layer What are the primary goals for DQI/NAL? • To support the efficient communication between the nodes of the parallel component of the ARCHIPEL database system • To organize the structure of this layer in such a way that it can be implemented on additional network hardware with little additional effort Januari 29th, 1999 University of Amsterdam slide 2

3. Target platform characteristics Cluster of server machines connected by a high-speed SAN/LAN network typical high-speed interconnect • commodity equipment such as switched Fast Ethernet, SCI, Myrinet, switched Gigabit Ethernet with a high level of reliability typical server • multi-processor machine build with off-the-shelve components (PCI-based, EIDE/SCSI hard disks, often Intel processors) capable of running a variety of operating systems Example: Januari 29th, 1999 University of Amsterdam slide 3

4. Application domain characteristics • number of database nodes together form one parallel database • threads within the different db-node processes have to communicate with each other • facilities for thread-to-thread communication must offer reliable, in-order transmission • thread-to-thread connections may carry short messages but also long streams of data • thread-to-thread connections are setup and broken down very frequently • every db-node is offering a variety of services for both external clients and internal clients (other db-nodes) to support functions like parallel query processing, parallel transaction management, replication, etc. Januari 29th, 1999 University of Amsterdam slide 4

5. Requirements for communication layer Hardware interfacing • utilizing the network hardware features (flow control, error detection) and available low-level interfaces to the network hardware as good as possible • easily enable the implementation on additional network hardware Application interfacing • support a single addressing scheme based on process identifier • support the efficient setup and brake-down of thread-to-thread connections • must be threadsafe Performance • focus on both bandwidth and latency • minimizing buffering and memory to memory copying • scale on multi-processor machines Januari 29th, 1999 University of Amsterdam slide 5

6. Existing implementations versus own implementation Existing communication layer implementations • combination of threadsafeness and specific requirements for the application interfacing is not available Own implementation • specifically addresses the needs imposed by the application: no redundant functionality, such as with MPI, resulting in less complex code • fits more precisely to the existing/available network hardware: this is an advantage for projects such as ARCHES (Gigabit Ethernet technology) • takes more effort Januari 29th, 1999 University of Amsterdam slide 6

7. Global design Network Abstraction Layer • providing basic reliable, in-order transmission of messages between processes • responsible for setup and shutdown of network connection between processes: addressing of messages is based on process identifier Descriptor Queue Interface: • providing communication primitives to the application by using the functionality offered by the NAL Januari 29th, 1999 University of Amsterdam slide 7

8. Network Abstraction Layer Objectives • Provide a generic network interface that can be implemented on a wide range of different kinds of network hardware • Support the DQI layer as good as possible: reliable, in-order transmission Interface • Setup, shutdown, etc: NALInitNetwork(), NALShutdownNetwork(), NALProcId(), NALNProcs(), NALMaxMessageSize() • Blocking send, blocking receive, poll: NALSend(), NALRecv(), NALPoll() Reference implementation : NALTCP • using TCP/IP Januari 29th, 1999 University of Amsterdam slide 8

9. Descriptor Queue Interface [1/2] • Based on concepts of U-Net (Cornell University) and VIA • Descriptor: identifies connections and describes buffer to send from/receive in • Connection: {local process id, local port number, remote process id, remote port number} • Addressing scheme much as with UDP: no explicit setup/shutdown of connection Januari 29th, 1999 University of Amsterdam slide 9

10. Descriptor Queue Interface [2/2] Interface • setup, shutdown, etc.: DQICreateDQ(), DQIDestroyDQ(), DQIProcId(), DQINProcs(), DQIMaxMessageSize() • port operations: DQIReserveSystemPort(), DQIReservePort(), DQIReleaseSystemPort(), DQIReleasePort() • actions on descriptors (blocking): DQIPostSend(), DQISendWait(), DQIPostRecv(), DQIRecvWait() Implementation uses pthreads Januari 29th, 1999 University of Amsterdam slide 10

11. Possible improvements / future work Improvements • send/receive buffer memory allocation through the NAL: easier implementation of NAL for network devices that support DMA access • performance tuning of DQI (look into redundant blocking) Future work • NAL implementation directly on top of Gigabit Ethernet (also ARCHES project) • NAL implementation on other network hardware Januari 29th, 1999 University of Amsterdam slide 11