1 / 17

SC’13 BoF Discussion

SC’13 BoF Discussion. Sean Hefty Intel Corporation. SC’13 BoF Feedback. Most people I’ve spoken with are highly supportive of the concepts being discussed General alignment Active discussion at SC ‘13 Engagement is high Good participation Want even wider focus outside of MPI

damali
Download Presentation

SC’13 BoF Discussion

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SC’13 BoF Discussion Sean Hefty Intel Corporation

  2. SC’13 BoF Feedback • Most people I’ve spoken with are highly supportive of the concepts being discussed • General alignment • Active discussion at SC ‘13 • Engagement is high • Good participation • Want even wider focus outside of MPI • Diverse viewpoints helping to drive conversation • More likely to produce better product • Need to understand all concerns • Address as many concerns as reasonable www.openfabrics.org

  3. Main Concerns from BoF • Application centric APIs may require new protocols • Must ensure wire compatibility • RDMA CM provides a simpler abstraction over verbs • How does this effort differ • Some applications want low-level access • Technology specific apps www.openfabrics.org

  4. Evolving OFA Software from the Viewpoint of libibverbs • Analyze libibverbs and librdmacm details • Selected issues • Previously covered fast path call overhead • Identify specific areas for improvement • Abstract the improvements to create a higher-performing, more extensible framework www.openfabrics.org

  5. Current libraries librdmacm Verb Helpers Infiniband verbs interface libibverbs Verbs Provider User to kernel ‘command’ interface www.openfabrics.org

  6. libibverbs Infiniband verbs interface libibverbs Verbs Provider User to kernel ‘command’ interface www.openfabrics.org

  7. libibverbs Structures • structibv_qp • pthread objects(mutex, cond) • internal counter • internal handle • qp data(state, qpn, type) • 5 data pointers • user context • structibv_cq • pthread objects (mutex, cond) • internal counters • internal handle • 2 data pointers • user context Exposes internal implementation • IB specific QP data • Not useful for iWarp • Not needed with rdma cm Data pointers may not be valid or needed by app www.openfabrics.org

  8. libibverbs Structures • structibv_context • structibv_device *device • structibv_context_ops ops; • intcmd_fd; • … Provider implemented functions • Embedded structure – fixed size! • Adding new calls is non-trivial • No mechanism to do so upstream • Provider functions associated with an opened device • Cannot optimize call based on use • Branches in code path and larger data structures www.openfabrics.org

  9. librdmacm No-op for iwarp and RoCE Scalability issues • Original APIs • Create ID, resolve address, resolve route • Connect, listen, accept, reject • Newer APIs • Getaddrinfo, create endpoint • Verb helpers • Register memory • Send/receive, rdma read/write, UD send Required for UD Addresses scalability and ease of use Simplified calls, but limited by verbs API www.openfabrics.org

  10. Evolving ‘Verbs’ • Merge libibverbs and librdmacm • Eliminate duplicate functionality • Remove internal implementation details from the API • Abstract data structures • Associate functionality with specific objects • Enable optimized code paths • Abstract objects for extensibility • Enable other use cases www.openfabrics.org

  11. BoF Concern • Application centric APIs may require new protocols • Must ensure wire compatibility • Transport protocol is exposed by framework • API definitions and protocol definitions are separate • Vendor-specific protocols are already being defined outside of verbs • PSM, MXM, FCA • TBD: discuss related protocols • Address resolution, route lookup, connection setup www.openfabrics.org

  12. BoF Concern • RDMA CM provides a simpler abstraction over verbs • How does this effort differ • RDMA CM abstractions are limited by functions exported by verbs • Still filter to single post send/recv call • RDMA CM is highly dependent on, but maintained separately • RDMA CM use is required for iWarp • RDMA CM functionality is carried forward www.openfabrics.org

  13. BoF Concern • Some applications want low-level access • Technology specific apps • Low-level access will still be possible • Focus is on ease of use (librdmacm model) with high-performance (libibverbs fast path) www.openfabrics.org

  14. (Scalable) Fabric Interfaces Fabric Interfaces Q: What is implied by incorporating interface sets under a single framework? Objects exist that are usable between the interfaces Isolated interfaces turn the framework into a complex dlopen Interfaces are composable May be used together Control Interface Atomics Message Queue RDMA Collective Operations Active Messaging CM Services Tag Matching www.openfabrics.org

  15. Migrating Providers from Verbs to FI RDMA Message Queue CM Services FI libfabric Verbs ibverbs abstraction layer libibverbs RDMA CM Verbs Provider Verbs Provider www.openfabrics.org

  16. Migrating Apps from Verbs to FI • Expose ‘verbs’ interfaces directly from FI • Use macros to convert ‘libibverbs’ exported calls to FI calls • Or layer libibverbs over libfabric • Applications must recompile • Minimal benefit to app • Dependent on accessed data structures and interfaces www.openfabrics.org

  17. Migrating Apps from Verbs to FI • Define ‘verbs’ compatibility mode • Allows mapping objects between interfaces • E.g. QP fabric socket CQ  EC • Restricts implementation • Mapping must be documented • Identified by protocol • Allow software to adopt new interfaces selectively • E.g. send/recv/ec_read FI verbs libibverbs libfabric Dual-Provider Library Verbs Provider FI Provider www.openfabrics.org

More Related