1 / 27

Module 5 Fabrics

Module 5 Fabrics. <Place supporting graphic here>. Module Objectives:. Overview of the Network Daughter Card (NDC). Fabrics and Mezzanine discussion. Specifically, Fabrics A, B, and C along with the communication differences between quarter, half and full-height blades.

petra
Download Presentation

Module 5 Fabrics

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Module 5 Fabrics <Place supporting graphic here>

  2. Module Objectives: • Overview of the Network Daughter Card (NDC). • Fabrics and Mezzanine discussion. Specifically, Fabrics A, B, and C along with the communication differences between quarter, half and full-height blades. • A brief overview of Flexaddress.

  3. Network Daughter Card (NDC) OverviewFabrics and Port Mapping

  4. Network Daughter CardsBlade Server NDC The 12G blade Network Daughter Card (NDC) is a custom form-factor card that contains a complete NIC subsystem. The PowerEdge blade server systems support NDC cards instead of the traditional LAN on Motherboard (LOM) design . Some blade server models can house up to two NDCs. The feature is marketed as a Dell Select NIC Adapter. The NDC has the typical features and behavior of a traditional LOM subsystem (including NC-SI management interface), but with the added benefit of flexibility by allowing customers to choose their favored network types, speed, and vendors. Another advantage of the NDC card is that it presents an easy upgrade path from 1 Gbps to 10 Gbps LAN speeds.

  5. Network Daughter Card (NDC) (continued)Choosing which NDC to use • New Mezzanine and Network Daughter Cards: • Broadcom 57810-K Dual-Port 10Gbit/s KR bNDC • Broadcom 57810-K Dual-Port 10Gbit/s KR Mezzanine Card • Intel Niantic 2x10GbE KR NDC • Intel 10 Gigabit KR/KX Dual-Port X520-KR2 bNDC • Intel Powerville 4x1GbE Mezzanine Card • QLogic QMD8262-k Dual-Port 10 Gigabit KR bNDC • QLogic QME8262-k Dual-Port 10 Gigabit KR Blade Mezzanine Card Typically, the system board will have one blade NDC connector for two-socket, half-height blades. Note: An NDC is always required for the system to POST without error.

  6. Networking- Fabric ChoiceDell Select Network Adapter Offerings for 12G (=>600 series rack/blade) Physical Layer Choice (rack only) Vendor Choice Interface Choice LOM Card Base-T (1GbE, 10GbE) SFP+ Fiber/Copper (10GbE, FCoE) Qlogic=Blade Only Technology Choice 1 Gigabit 10 Gigabit • 2x 10GbE (blade) • 2x10GbE + 2x1GbE (Rack) • 4x1GbE (rack) • 4x10GbE (future) FCoE (CNA)-All 10Gb NIC Partitioning (Qlogic & Broadcom) iSCSI Offload (Broadcom & Qlogic) SR-IOV (All 10Gb)

  7. Fabric A LOM / Network Daughter Card (NDC) • Regardless of speed Fabric A is always the onboard NICs (LOM / 1GbE or 10GbE). • With the KR (v1.1) midplane, the customer can use 10GbE LOMs, now in the form of Network Daughter Cards (NDC). • The NDC provides the customer vendor flexibility. • Fabric A = Still Ethernet Only. • An NDC is always required for the system to POST without error

  8. Blade Mezzanine Cards • All M-series blades, with the exception of the M420, have two or four mezzanine card slots allowing you to install mezzanine cards to communicate with fabrics B and C. • For the blade to boot, the mezzanine card must match the fabric as determined by the I/O module (IOM) installed in the enclosure. • 10Gb Ethernet. • 1 Gb Ethernet. • Fibre Channel. • InfiniBand.

  9. Blade Mezzanine Cards • Dual-port mezzanine cards map two ports/paths – one to each of the redundant I/O modules (eg. B1 and B2) providing high availability. • Quad-port mezzanine cards map four ports – two to each of the redundant I/O modules, providing added bandwidth and high availability.

  10. PowerEdge M1000e Chassis Fabrics and Capabilities Fabric A1Reserved for 1/10GbE LOMs or Select Network Adapters Fabric A2Reserved for 1/10GbE LOMs or Select Network Adapters A1 B1 C1 C2 B2 A2 Fabric B11/10/40 GbE, 4/8/16Gb FC, 20/40/56GbIB Fabric B21/10/40 GbE, 4/8/16Gb FC, 20/40/56GbIB Fabric C11/10/40 GbE, 4/8/16Gb FC, 20/40/56Gb IB Fabric C21/10/40 GbE, 4/8/16Gb FC, 20/40/56Gb IB Colors chosen to facilitate whiteboard discussions. The capabilities of the enhanced midplane (1.1) are shown above

  11. B1 B2 B1 B2 B1 B2 B1 B2 M-Series Blade I/O Fabrics C1 C2 C1 C2 C1 C2 C1 C2 • Quarter Height Blades • One dual port LOM • IOM with 32 internal ports (M6348 or Dell Force10 MXL) is needed to connect all LOM ports on all blades • 2 x 32 port IOMs needed to connect the 2 LOM ports on each blade • One fabric B OR fabric C mezzanine card • Half Height Blades • One Select Network Adapter or LOM • One fabric B mezzanine card • One fabric C mezzanine card • Full Height Blades • Two Select Network Adapters or LOMs • Two fabric B mezzanine cards • Two fabric C mezzanine cards OR Quarter Height Half Height Full Height

  12. M1000e Midplane Mapping and Capabilities Right side of chassis Fabric A Capabilities: M1000e Mid-plane Half-Height Blade Slot (1 of 16) Fabric A2Ethernet I/O Module • Up to 2 lanes to each IOM • 1Gb or 10Gb Ethernet per each lane Fabric ALOM Fabric BMezzanine Fabric B2I/O Module Fabric CMezzanine Fabric C2I/O Module Fabric B & C Capabilities: • Up to 4 lanes to each IOM • 1Gb or 10Gb Ethernet per each lane or 40Gb Ethernet using all 4 lanes • 4Gb, 8Gb, or 16Gb Fibre Channel over 1 lane to each IOM • 40Gb QDR, 40Gb FDR10, or 56Gb FDR InfiniBand using all 4 lanes. 20Gb DDR InfiniBand using 2 lanes. 2 lanes 2 lanes 4 lanes 4 lanes 2 lanes 4 lanes 4 lanes 4 lanes 4 lanes 2 lanes 4 lanes 4 lanes Front of Chassis (Rear of Chassis) External Fabric Connections (Rear of Chassis) Fabric C1I/O Module Half-Height Blade Slot (16 of 16) Fabric ALOM Fabric B1I/O Module Fabric BMezzanine Fabric A1Ethernet I/O Module Fabric CMezzanine A lane represents a single link between an adapter and an IOM. Each port will utilize 1, 2 or 4 lanes depending on the communication protocol. 8.4Tb Mid-plane Left side of chassis

  13. I/O Fabric Architecture for Half-Height Blades Right side of chassis Fabric A: M1000e Mid-plane Half-Height Blade Server (1 of 16) Fabric A2Ethernet I/O Module • Dual port and Quad port 1Gb or 10Gb Ethernet adapters Fabric ALOM CPU 1 4-8 lane PCIe Fabric BMezzanine Fabric B2I/O Module 8 lane PCIe CPU 2 Fabric CMezzanine 8 lane PCIe Fabric C2I/O Module Fabric B & C: • Ethernet, Fibre Channel, &/or InfiniBand mezzanine cards • Dual port 1Gb and 10Gb Ethernet mezzanine cards • Quad port 1Gb Ethernet mezz. and capable of quad port 10Gb Ethernet mezzanine • Dual port Fibre Channel mezz. • Dual port InfiniBand mezzanine Front of Chassis (Rear of Chassis) External Fabric Connections (Rear of Chassis) Fabric C1I/O Module Half-Height Blade Server (16 of 16) Fabric ALOM Fabric B1I/O Module CPU 1 4-8 lane PCIe Fabric BMezzanine 8 lane PCIe CPU 2 Fabric A1Ethernet I/O Module Fabric CMezzanine 8 lane PCIe Link between a dual port adapter and switch of same fabric type Additional link provided by quad-port adapter cards and an IOM with 32 internal ports 8.4Tb Mid-plane Left side of chassis

  14. Port Mapping of Half Height blades with Dual Port Adapters to IOMs with 16 or 32 Internal Ports IOM ports mapped to half height blade slots • All six IOMs have the same port mapping for half height blades • IOMs with 32 internal ports will only connect with 16 internal ports when using dual port adapters Slot1 Slot2 Slot3 Slot4 Slot5 Slot6 Slot7 Slot8 IOM1 P1 IOM1 P2 IOM1 P3 IOM1 P4 IOM1 P5 IOM1 P6 IOM1 P7 IOM1 P8 IOM2 P1 IOM2 P2 IOM2 P3 IOM2 P4 IOM2 P5 IOM2 P6 IOM2 P7 IOM2 P8 A2,B2,C2 A1,B1,C1 IOM2 IOM1 Slot9 Slot10 Slot11 Slot12 Slot13 Slot14 Slot15 Slot16 IOM1 P9 IOM1 P10 IOM1 P11 IOM1 P12 IOM1 P13 IOM1 P14 IOM1 P15 IOM1 P16 IOM2 P15 IOM2 P9 IOM2 P10 IOM2 P11 IOM2 P12 IOM2 P13 IOM2 P14 IOM2 P16

  15. Port Mapping of Half Height blades with Quad Port Adapters to IOMs with 32 Internal Ports IOM ports mapped to half height blade slots • An IOM with 32 internal ports is required to connect to all quad port adapters • All six IOMs have the same port mapping for half height blades Slot1 Slot2 Slot3 Slot4 Slot5 Slot6 Slot7 Slot8 IOM1 P1 IOM1 P2 IOM1 P3 IOM1 P4 IOM1 P5 IOM1 P6 IOM1 P7 IOM1 P8 IOM2 P1 IOM2 P2 IOM2 P3 IOM2 P4 IOM2 P5 IOM2 P6 IOM2 P7 IOM2 P8 IOM1 P17 IOM1 P18 IOM1 P19 IOM1 P20 IOM1 P21 IOM1 P22 IOM1 P23 IOM1 P24 A2,B2,C2 A1,B1,C1 IOM2 P17 IOM2 P18 IOM2 P19 IOM2 P20 IOM2 P21 IOM2 P22 IOM2 P23 IOM2 P24 IOM2 IOM1 Slot9 Slot10 Slot11 Slot12 Slot13 Slot14 Slot15 Slot16 IOM1 P9 IOM1 P10 IOM1 P11 IOM1 P12 IOM1 P13 IOM1 P14 IOM1 P15 IOM1 P16 IOM2 P15 IOM2 P9 IOM2 P10 IOM2 P11 IOM2 P12 IOM2 P13 IOM2 P14 IOM2 P16 IOM1 P25 IOM1 P26 IOM1 P27 IOM1 P28 IOM1 P29 IOM1 P30 IOM1 P31 IOM1 P32 IOM2 P25 IOM2 P26 IOM2 P27 IOM2 P28 IOM2 P29 IOM2 P30 IOM2 P31 IOM2 P32

  16. I/O Fabric Architecture for Full-Height Blades Right side of chassis Fabric A: M1000e Mid-plane Half-Height Blade Server (1 of 8) Fabric A2Ethernet I/O Module • Ethernet Only • Dual port 1Gb and 10Gb Ethernet adapters • Quad port 1Gb Ethernet and capable of quad port 10Gb Ethernet adapters Fabric ALOM CPU 1 4-8 lane PCIe Fabric BMezzanine Fabric B2I/O Module 8 lane PCIe CPU 2 Fabric CMezzanine 8 lane PCIe Fabric C2I/O Module Fabric B & C: • Ethernet, Fibre Channel, &/or InfiniBand mezzanine cards • Dual port 1Gb and 10Gb Ethernet mezzanine cards • Quad port 1Gb Ethernet mezz. and capable of quad port 10Gb Ethernet mezzanine • Dual port Fibre Channel mezz. • Dual port InfiniBand mezzanine Front of Chassis (Rear of Chassis) External Fabric Connections (Rear of Chassis) Fabric C1I/O Module Fabric ALOM Fabric B1I/O Module CPU 3 4-8 lane PCIe Fabric BMezzanine 8 lane PCIe CPU 4 Fabric A1Ethernet I/O Module Fabric CMezzanine 8 lane PCIe Link between a dual port adapter and switch of same fabric type Additional link provided by quad-port adapter cards and an IOM with 32 internal ports 8.4Tb Mid-plane Left side of chassis

  17. Port Mapping of Full Height blades with Quad Port Adapters to IOMs with 32 Internal Ports IOM ports mapped to full height blade slots • All six IOMs have the same port mapping for half height blades • An IOM with 32 internal ports is required to connect to all quad port adapters Slot1 Slot2 Slot3 Slot4 Slot5 Slot6 Slot7 Slot8 IOM1 P1 IOM1 P2 IOM1 P3 IOM1 P4 IOM1 P5 IOM1 P6 IOM1 P7 IOM1 P8 IOM2 P1 IOM2 P2 IOM2 P3 IOM2 P4 IOM2 P5 IOM2 P6 IOM2 P7 IOM2 P8 IOM1 P17 IOM1 P18 IOM1 P19 IOM1 P20 IOM1 P21 IOM1 P22 IOM1 P23 IOM1 P24 A2,B2,C2 A1,B1,C1 IOM2 P17 IOM2 P18 IOM2 P19 IOM2 P20 IOM2 P21 IOM2 P22 IOM2 P23 IOM2 P24 IOM2 IOM1 IOM1 P9 IOM1 P10 IOM1 P11 IOM1 P12 IOM1 P13 IOM1 P14 IOM1 P15 IOM1 P16 IOM2 P15 IOM2 P9 IOM2 P10 IOM2 P11 IOM2 P12 IOM2 P13 IOM2 P14 IOM2 P16 IOM1 P25 IOM1 P26 IOM1 P27 IOM1 P28 IOM1 P29 IOM1 P30 IOM1 P31 IOM1 P32 IOM2 P25 IOM2 P26 IOM2 P27 IOM2 P28 IOM2 P29 IOM2 P30 IOM2 P31 IOM2 P32

  18. I/O Fabric Architecture with Quarter Height Blades Right side of chassis Fabric A: Quarter-Height (Slot a) M1000e Mid-plane Fabric A2Ethernet I/O Module • Dual port 10Gb Ethernet LOM • Connectivity for both LOM ports requires IOMs with 32 internal ports • Two IOMs with only 16 internal ports will only provide a connected to a single LOM port on each blade Fabric ALOM CPU 1 8 lane PCIe CPU 2 Fabric CMezzanine 8 lane PCIe Fabric B2I/O Module Quarter-Height (Slot b) Fabric ALOM CPU 1 8 lane PCIe Fabric C2I/O Module CPU 2 Fabric BMezzanine 8 lane PCIe Fabric B & C: Front of Chassis (Rear of Chassis) External Fabric Connections (Rear of Chassis) • Ethernet, Fibre Channel, &/or InfiniBand mezzanine cards • Each quarter height blade only has one mezzanine card Quarter-Height (Slot c) Fabric C1I/O Module Fabric ALOM CPU 1 8 lane PCIe CPU 2 Fabric CMezzanine 8 lane PCIe Fabric B1I/O Module Quarter-Height (Slot d) Fabric ALOM CPU 1 8 lane PCIe Link between a dual port adapter and switch of same fabric type Additional link provided by quad-port adapter cards and an IOM with 32 internal ports Redundant LOM link that requires an IOM with 32 internal ports. There will be no connection on this link with IOMs with only 16 ports Fabric A1Ethernet I/O Module CPU 2 Fabric BMezzanine 8 lane PCIe 8.4Tb Mid-plane Left side of chassis

  19. PowerEdge Blade Server M420Fabric Mapping – Quarter-height

  20. Port Mapping of Quarter Height blades to two IOMs with 32 Internal Ports on Fabric A: Full LOM Port Redundancy IOM ports mapped to quarter height blade slots • On fabric A, two IOMs with 32 internal ports provide connectivity to two ports of the LOM on each quarter height blade. • Full LOM port redundancy Slot1a Slot2a Slot3a Slot4a Slot5a Slot6a Slot7a Slot8a A1 P1 A1 P2 A1 P3 A1 P4 A1 P5 A1 P6 A1 P7 A1 P8 A2 P17 A2 P18 A2 P19 A2 P20 A2 P21 A2 P22 A2 P23 A2 P24 Slot1b Slot2b Slot3b Slot4b Slot5b Slot6b Slot7b Slot8b A1 P17 A1 P18 A1 P19 A1 P20 A1 P21 A1 P22 A1 P23 A1 P24 IOM A1 and A2 A2 P1 A2 P2 A2 P3 A2 P4 A2 P5 A2 P6 A2 P7 A2 P8 A1 A2 Slot1c Slot2c Slot3c Slot4c Slot5c Slot6c Slot7c Slot8c A1 P9 A1 P10 A1 P11 A1 P12 A1 P13 A1 P14 A1 P15 A1 P16 A2 P25 A2 P26 A2 P27 A2 P28 A2 P29 A2 P30 A2 P31 A2 P32 Slot1d Slot2d Slot3d Slot4d Slot5d Slot6d Slot7d Slot8d A1 P25 A1 P26 A1 P27 A1 P28 A1 P29 A1 P30 A1 P31 A1 P32 A2 P9 A2 P10 A2 P11 A2 P12 A2 P13 A2 P14 A2 P15 A2 P16

  21. Port Mapping of Quarter Height blades to two IOMs with 16 Internal Ports on Fabric A: No LOM Port Redundancy IOM ports mapped to quarter height blade slots • On fabric A, two IOMs with 16 internal ports provide connectivity to one port of the LOM on each quarter height blade. • Connectivity but not redundancy (only 1 LOM port per blade is connected) Slot1a Slot2a Slot3a Slot4a Slot5a Slot6a Slot7a Slot8a A1 P1 A1 P2 A1 P3 A1 P4 A1 P5 A1 P6 A1 P7 A1 P8 Slot1b Slot2b Slot3b Slot4b Slot5b Slot6b Slot7b Slot8b IOM A1 and A2 A2 P1 A2 P2 A2 P3 A2 P4 A2 P5 A2 P6 A2 P7 A2 P8 A1 A2 Slot1c Slot2c Slot3c Slot4c Slot5c Slot6c Slot7c Slot8c A1 P9 A1 P10 A1 P11 A1 P12 A1 P13 A1 P14 A1 P15 A1 P16 Slot1d Slot2d Slot3d Slot4d Slot5d Slot6d Slot7d Slot8d A2 P9 A2 P10 A2 P11 A2 P12 A2 P13 A2 P14 A2 P15 A2 P16

  22. Port Mapping of Quarter Height blades to four IOMs on Fabric B&C: Full Mezz Card Port Redundancy IOM ports mapped to quarter height blade slots • On fabric B&C, four IOMs provide full redundancy (connect all ports) to all mezzanine cards. Slot1a Slot2a Slot3a Slot4a Slot5a Slot6a Slot7a Slot8a C1 P1 C1 P2 C1 P3 C1 P4 C1 P5 C1 P6 C1 P7 C1 P8 C2 P1 C2 P2 C2 P3 C2 P4 C2 P5 C2 P6 C2 P7 C2 P8 Slot1b Slot2b Slot3b Slot4b Slot5b Slot6b Slot7b Slot8b B1 P1 B1 P2 B1 P3 B1 P4 B1 P5 B1 P6 B1 P7 B1 P8 IOM B1,B2, C1 and C2 B2 P1 B2 P2 B2 P3 B2 P4 B2 P5 B2 P6 B2 P7 B2 P8 B2 B1 C1 C2 Slot1c Slot2c Slot3c Slot4c Slot5c Slot6c Slot7c Slot8c C1 P15 C1 P16 C1 P13 C1 P14 C1 P9 C1 P10 C1 P11 C1 P12 C2 P9 C2 P10 C2 P11 C2 P12 C2 P13 C2 P14 C2 P15 C2 P16 Slot1d Slot2d Slot3d Slot4d Slot5d Slot6d Slot7d Slot8d B1 P9 B1 P10 B1 P11 B1 P12 B1 P13 B1 P14 B1 P15 B1 P16 B2 P15 B2 P16 B2 P13 B2 P14 B2 P9 B2 P10 B2 P11 B2 P12

  23. FlexAddress

  24. FlexAddress • The CMC offers simple interface for enabling FlexAddress by chassis, by slot, or by fabric, assigning WWN/MAC values in place of factory-assigned WWN/MAC • User-configurable enablement of iSCSI MAC, Ethernet MAC, and/or WWN Persistence which allows blades to be swapped without affecting SAN Zoning, iSCSI zoning, or any MAC-dependent functions

  25. FlexAddress - Secure Digital (SD) card • Data for enabling FlexAddress is stored on a pluggable Secure Digital (SD) card. • The CMC will validate the SD card and its contents before enabling FlexAddress functions. • Data is read from an SD Card and is stored in the I/O Control Panel for future use and reference.

  26. Deploying FlexAddress • Log into the CMC GUI and Click on and expand the Chassis Overview located to the left hand side of the screen. • To enable FlexAddress, select Server overview> setup> FlexAddress and you will see the Flex Address Fabric Configuration for Fabrics A, B and C. • Validate that FlexAddress is enabled by clicking on Server Overview>Properties> Status.

  27. Questions?

More Related