Kanoj sarcar sr staff engineer sun microsystems presented by bill strahm
This presentation is the property of its rightful owner.
Sponsored Links
1 / 8

Solaris IPoIB (ibd) Implementation PowerPoint PPT Presentation


  • 101 Views
  • Uploaded on
  • Presentation posted in: General

Kanoj Sarcar Sr Staff Engineer Sun Microsystems (presented by Bill Strahm). Solaris IPoIB (ibd) Implementation. Architecture Overview. DLPI driver Only supported for GID0 Interop (TopSpin, openib) Software structure: IP | SDP | uDAPL | ibd | | | ...

Download Presentation

Solaris IPoIB (ibd) Implementation

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Kanoj sarcar sr staff engineer sun microsystems presented by bill strahm

Kanoj Sarcar

Sr Staff Engineer

Sun Microsystems

(presented by Bill Strahm)

Solaris IPoIB (ibd) Implementation


Architecture overview

Architecture Overview

  • DLPI driver

    • Only supported for GID0

  • Interop (TopSpin, openib)

  • Software structure:

    IP | SDP | uDAPL |

    ibd | | | ...

    |IBTF / IBMF |

    | HCA driver |

  • Performance


Tools changes 20 byte mac

Tools changes (20 byte MAC)

  • New SIOC*XARP ioctls and “struct xarpreq”

    • arp(1M) changes

    • PPPD proxyarp etc

  • Undocumented helper function ala ether_ntoa, ether_aton

    • extern char *_link_ntoa(const unsigned char *, char *, int, int);

    • extern unsigned char *_link_aton(const char *, int *);

  • ifconfig(1M), snoop(1M)

  • RARP

    • /etc/ethers, NIS assume 6 bytes!


Tools changes ipoib specific

Tools changes (IPoIB specific)

  • DHCP agent

    • Usage of BROADCAST bit

    • Setting IB hardware type in DHCP packet in htype

    • Client id format

  • Code Len Type |<---------------- Client-Identifier -------------->|

  • +-----+-----+-----+-----+-----+-----+-----+-------------------....----+

  • | 61 | 21 | 00 | 00 (4 octets) | 16 octet GID |

  • +-----+-----+-----+-----+-----+-----+-----+-------------------....----+

  • DHCP Server

    • No explicit ARP cache setting with DHCP chaddr

  • Snoop(1M) parses IPoIB, disallows MAC filters


Tools changes ipoib specific part ii

Tools changes (IPoIB specific)Part II

  • ifconfig(1M)

  • hme0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2

  • inet 10.6.98.56 netmask ff000000 broadcast 10.255.255.255

  • ether 0:3:ba:24:4:df

  • ibd1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 2044 index 4

  • inet 192.168.100.100 netmask ffffff00 broadcast 192.168.100.255

  • ipib 0:0:4:a:0:0:0:0:0:0:12:34:0:2:c9:1:9:76:57:11

  • SNMP (net-snmp)

    • ifSpeed consistent with draft-ietf-ipoib-ibif-mib-07.txt

    • ifPhysAddress as MAC address (to match ARP)

    • octet/packet counts only for ipoib network interface


Ipoib oddities

IPoIB oddities

  • DLPI receiver indicates “src addr unavailable” for unicast packets

    • If transmitter guarantees GID0 usage, SM assisted reverse lookup possible

  • RARP unsupported

    • Non persistant MAC

    • Server requires client's MAC address for 3rd party lookup


Ipoib configuration

IPoIB Configuration

  • One time discovery

    • Reconfiguration boot or IO discovery command

    • Create possible ipoib instance per port/Pkey

    • SM must be up

  • Interface initialization

    • IP “plumb” triggers MCG probe as validity check

    • SM must be up, fabric admin must create IP MCG

  • Decisions

    • End nodes do not create IP MCG

    • End nodes do not have to know IB parameters

  • Issues

    • IP multicast group membership requests

    • Can not wait, can not spool!

    • Solaris IP stack joins 224.0.0.1 at interface init


Interesting driver details

Interesting Driver Details

  • QPN/CQ/WQE/PathRecord

  • Tx path copy vs registration

  • GID/QPN to Pathrecord (hash/cache)

  • Disallow (DLPI) change MAC address

  • Async thread for (blocking) SM communication

  • IB link up/down handling

    • ULP notification for IPMP failover

  • MCG create/delete trap handling

  • 2 MCG membership lists (full/sendonly, non)

  • MCG pathrecord and membership implication


  • Login