1 / 29

Blue Gene/P System Overview - Hardware

This overview provides information on the hardware specifications and components of the Blue Gene/P and Blue Gene/L systems, including racks, node cards, compute cards, and networking capabilities.

wlori
Download Presentation

Blue Gene/P System Overview - Hardware

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Blue Gene/PSystem Overview - Hardware

  2. 72 Racks, 72x32x32 Cabled 8x8x16 Rack BlueGene/P Racks 32 Node Cards 1 PF/s 144 TB Node Card (32 chips 4x4x2) 32 compute, 0-2 IO cards 13.9 TF/s 2 TB Compute Card 1 chip, 20 DRAMs 435 GF/s 64 GB Chip 4 processors 13.6 GF/s 2.0 GB DDR2 13.6 GF/s 8 MB EDRAM Intro (BG/P and BG/L)

  3. System 64 Racks, 64x32x32 Rack Cabled 8x8x16 32 Node Cards Node Card (32 chips 4x4x2) 16 compute, 0-2 IO cards 180/360 TF/s 32 TB Processor Card 2 chips, 1x2x1 2.8/5.6 TF/s 512 GB Node Dual Processor 90/180 GF/s 16 GB 5.6/11.2 GF/s 1.0 GB Intro (BG/P and BG/L) BlueGene/LRacks

  4. Comparison of BG/L and BG/P nodes

  5. Comparison of BG/L and BG/P nodes (2)

  6. Blue Gene/P Rack

  7. Blue Gene/L Rack

  8. BG/P and BG/L Cooling

  9. BG/P Node Card 32 Compute nodes Local DC-DC regulators (6 required, 8 with redundancy) Optional IO card (one of 2 possible) with 10Gb optical link

  10. Blue Gene/L HardwareNode Card

  11. BG/P Compute Card

  12. Three type of compute/IO cards • 1.07 volts 44V3572 • 1.13 volts 44V3575 • 1.19 volts 44V3578 • Cannot be mixed within a node card.

  13. Blue Gene/L Processor Card

  14. Blue Gene/P HardwareLink Cards

  15. Blue Gene/L HardwareLink Cards

  16. Blue Gene/P Hardware Service Card

  17. Blue Gene/L HardwareService Card

  18. Naming conventions (1) Racks: Rxx Midplanes: Rxx-Mx Rack Column (0-F) Rack Row (0-F) See picture below for details Midplane (0-1) 0=Bottom 1=Top Rack Column (0-F) Rack Row (0-F) Bulk Power Supply: Rxx-B Clock Cards: Rxx-K Bulk Power Supply Rack Row (0-F) Rack Column (0-F) Clock Rack Column (0-F) Rack Row (0-F) Fan Assemblies: Rxx-Mx-Ax Power Cable: Rxx-B-C Fan Assembly (0-9) 0=Bottom Front, 4=Top Front 5=Bottom Rear, 9=Top Rear Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F) Power Cable Bulk Power Supply Rack Row (0-F) Rack Column (0-F) Fans: Rxx-Mx-Ax-Fx Power Modules: Rxx-B-Px Fan (0-2) 0=Tailstock 2= Midplane Fan Assembly (0-9) 0=Bottom Front, 4=Top Front 5=Bottom Rear, 9=Top Rear Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F) Power Module (0-7) 0-3 Left to right facing front 4-7 left to right facing rear Bulk Power Supply Rack Row (0-F) Rack Column (0-F)

  19. Naming conventions (2) Note: Master service card for a rack is always Rxx-M0-S Service Cards: Rxx-Mx-S Compute Cards: Rxx-Mx-Nxx-Jxx Compute Card (04 through 35) Node Card (00-15) Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F) Service Card Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F) Link Cards: Rxx-Mx-Lx 0=Bottom Front 1=Top Front 2=Bottom Rear 3=Top Rear See picture below for details Link Card (0-3) Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F) I/O Cards: Rxx-Mx-Nxx-Jxx I/O Card (00-01) Node Card (00-15) Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F) Node Cards: Rxx-Mx-Nxx 00=Bottom Front 07=Top Front 08=Bottom Rear 15=Top Rear Node Card (00-15) Midplane (0-1) 0-Bottom, 1=Top Rack Column (0-F) Rack Row (0-F)

  20. Rack Naming Convention Note: The fact that this illustration shows numbers 00 through 77 does not imply this is the largest configuration possible. The largest configuration possible is 256 racks numbered 00 through FF. Note direction of slant on covers Service card side

  21. Torus X-Y-Z Z Y X

  22. Node, Link, ServiceCard Names L1 N07 N06 N05 N04 S N03 N02 N01 N00 L0 L3 N15 N14 N13 N12 N11 N10 N09 N08 L2

  23. J32 J28 J24 J20 J16 J12 J08 J04 J33 J29 J25 J21 J17 J13 J09 J05 J00 Node Card J34 J30 J26 J22 J18 J14 J10 J06 J35 J31 J27 J23 J19 J15 J11 J07 J01 EN0 EN1

  24. Service Card Rack Row Indicator (0-F) Rack Column Indicator (0-F) Control Network Clock Input

  25. U00 Link Card U01 U02 U03 U04 U05 J00 J02 J04 J06 J08 J10 J12 J14 J01 J03 J05 J07 J09 J11 J13 J15

  26. Clock Card Output 9 Output 8 Output 7 Output 6 Slave Output 5 Master Output 4 Output 3 Output 2 Output 1 Output 0 Input

  27. Users Networks (BG/P and BG/L) File System Site Network Front-End Node Functional Network Service Node Blue Gene Core Administrator Service Network

  28. Blue Gene/P Interconnection Networks • 3 Dimensional Torus • Interconnects all compute nodes • Communications backbone for computations • Adaptive cut-through hardware routing • 3.4 Gb/s on all 12 node links (5.1 GB/s per node) • 0.5 µs latency between nearest neighbors, 5 µs to the farthest • MPI: 3 µs latency for one hop, 10 µs to the farthest • 1.7/2.6 TB/s bisection bandwidth, 188TB/s total bandwidth (72k machine) • Collective Network • Interconnects all compute and I/O nodes (1152) • One-to-all broadcast functionality • Reduction operations functionality • 6.8 Gb/s of bandwidth per link • Latency of one way tree traversal 2 µs, MPI 5 µs • ~62TB/s total binary tree bandwidth (72k machine) • Low Latency Global Barrier and Interrupt • Latency of one way to reach all 72K nodes 0.65 µs, MPI 1.6 µs • Other networks • 10Gb Functional Ethernet • I/O nodes only • 1Gb Private Control Ethernet • Provides JTAG access to hardware. Accessible only from Service Node system

  29. Blue Gene/L Networks • 3 Dimensional Torus • Interconnects all compute nodes (65,536) • Virtual cut-through hardware routing • 1.4Gb/s on all 12 node links (2.1 GB/s per node) • Communications backbone for computations • 0.7/1.4 TB/s bisection bandwidth, 67TB/s total bandwidth • Collective Network • One-to-all broadcast functionality • Reduction operations functionality • 2.8 Gb/s of bandwidth per link; Latency of tree traversal 2.5 µs • ~23TB/s total binary tree bandwidth (64k machine) • Interconnects all compute and I/O nodes (1024) • Ethernet IP • Incorporated into every node ASIC • Active in the I/O nodes (1:64) • All external comm. (file I/O, control, user interaction, etc.) • Control System Network • Boot, monitoring and diagnostics • Global Barrier and Interrupt

More Related