1 / 22

New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts

New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts. New Mexico Computing Applications Center HPC for New Mexico. 10 Gig-E. Overall Top-level Diagram. UNM Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack. Encanto Altix ICE System

mahon
Download Presentation

New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. New Mexico Computing Applications Center (NMCAC) System Diagrams & Physical Layouts

  2. New Mexico Computing Applications Center HPC for New Mexico

  3. 10 Gig-E Overall Top-level Diagram UNM Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack Encanto Altix ICE System 14,336 Compute Cores 1,792 Nodes 28 Racks New Mexico Tech Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack NMSU Altix ICE Exemplar 176 Compute Cores 22 Nodes 1 Rack

  4. Main System “Encanto” Overview • Major Components • 172 TFLOPS Altix ICE 8200 Cluster • 1792 Nodes, 3,584 Sockets, 14,336 Cores • 28.7 TB memory (2GB/core) • 172 TB Parallel File Subsystem (Lustre) • 20 OSS Service Nodes • 20 TB NAS & Archive Storage Subsystem • 1 System Admin Controller • 28 Rack Leader Controllers • 1 Batch Service Node • 5 Login Service Nodes • Lambda Rail Connectivity • 36 Racks – 28 Compute – 5 Storage – 3 Support

  5. IB DDR 10 Gig-E 20 4 10 Gig-E Encanto Top-level Diagram Lamba Rail Fiber 1 System Admin Controller Batch Node 1 1 Lambda Rail Gateway 2 Altix ICE 8200 Cluster 14,336 Compute Cores 1,792 Nodes 28 Racks 5 Qty 5 Login Service Nodes Login Service Nodes Login Service Nodes Login Service Nodes 33 IB Lines Login Service Nodes 5 20 1 Lustre-Based Parallel Storage Sub-system 172 TB Usable NAS CUBE / Archive Storage Sub-system NAS: 20 TB Usable Arch: 40TB Uncompressed

  6. NMCAC Qwest Fiber Connections to New Mexico “GigaPop” 505 Marquette Albuquerque, NM “Lambda Rail” Point of Presence 4100 Sara Road Rio Rancho, NM “Encanto” Supercomputer Existing National Lamba Rail Fiber Ring

  7. IB DDR 3 1 10 Gig-E Gig-E Encanto Lambda-Rail Router Diagram Lamba Rail Fiber Future Expansion Gateway/Router Cisco 6509 10-GigE Router 5 Login Service Node 2 1 1 Altix ICE 8200 Cluster 14,336 Compute Cores 1,792 Nodes 28 Racks Login Service Node 2 1 1 10 5 Login Service Node 2 1 1 Login Service Node 2 1 1 Login Service Node 2 1 1

  8. Login Node Login Node Qwest Geomax Qwest Geomax Login Node Login Node Login Node Encanto External Network Diagram 505 Marquette Site 4100 Sara Road Site Cisco 6513 Switch Cisco 6509 Switch Cisco 6509 Switch UNM NMTECH NMSU National Lamba Rail • UNM Switch • Owned by UNM • Connected to NLR • Connects to NMSU • Connects to NMTECH • Redundant DC Power • NMCAC Switch • Owned by NMCAC • Redundant DC Power • NMCAC Switch • Owned by NMCAC • Router between Fibre & Encanto • Redundant AC Power • Routing module to 5 Login Nodes • Load-leveling Module

  9. 20 10 Lustre Storage Units SAS FC4 Heartbeat IB DDR Gig-E Encanto Lustre-Based Storage System Meta-data Servers Altix ICE 8200 Cluster 14,336 Compute Cores 1,792 Nodes 28 Racks 6015B 1 IS220 1 6015B Place OSS XE310 servers on top of the following ICE racks (5, 6, 7, 8, 9, 10, 11, 12, 24, 25) Lustre Storage Unit 1 1 1 1 1 1 1 1 OSS OSS OSS OSS OSS OSS OSS OSS … XE310 Node XE310 Node XE310 Node XE310 Node XE310 Node XE310 Node XE310 Node XE310 Node IS4500 IS4500 IS4500 IS4500 OST OST OST OST

  10. 1 1 IB IB Altix XE 310 Node 2x3 GHz quad-core 16GB memory Altix XE 310 Node 2x3 GHz quad-core 16GB memory FC FC Loop Loop Loop Loop FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray Encanto Lustre Storage Unit Object Storage Servers 1 1 1 1 FC FC IS4500 Dual-controller 2GB Cache Object Storage Targets 64x 400GB 15K rpm Fibre Channel drives 12x ( 4 + 1 ) LUNs + 4 Hot Spares Across 4 FC Loops

  11. 1 1 1 1 IB IB IB IB FC4 FC4 FC4 FC4 FC4 FC4 FC4 FC4 FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC RAID Drive Tray FC4 Heartbeat IB DDR Gig-E Lustre Storage Cabling Details Altix XE 310 Node Altix XE 310 Node Altix XE 310 Node Altix XE 310 Node OSSs FC FC FC FC FC FC FC FC IS4500 Dual-controller IS4500 Dual-controller OSTs

  12. Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge FC FC FC Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge SATA RAID Drive Tray SATA RAID Drive Tray SATA RAID Drive Tray Tape Cartridge Tape Cartridge Tape Cartridge Tape Cartridge Combined NAS & Archive Storage Subsystem 4 IB Altix 450 Altix 450 Blade Enclosure Atempo Archive Server Altix 450 Blade Enclosure NAS Head Node NUMAlink FC FC FC FC Control Line Data Lines 1 1 1 2 FC FC FC FC SL500 Dual-LT03 Tape Drives IS4000 RAID-controller

  13. Altix ICE Rack 014 (Front) Altix ICE Rack 010 (Front) Altix ICE Rack 013 (Front) Altix ICE Rack 015 (Front) Altix ICE Rack 016 (Front) Altix ICE Rack 006 (Front) Altix ICE Rack 007 Altix ICE Rack 008 (Front) Altix ICE Rack 005 (Front) Altix ICE Rack 009 Altix ICE Rack 012 (Front) ADMIN Rack (Front) Altix ICE Rack 004 (Front) Altix ICE Rack 003 (Front) Altix ICE Rack 001 (Front) Altix ICE Rack 002 (Front) Altix ICE Rack 011 (Front) {Front) 026 Altix ICE Rack {Front) 019 Altix ICE Rack {Front) 028 Altix ICE Rack {Front) 020 Altix ICE Rack {Front) 025 Altix ICE Rack {Front) 017 Altix ICE Rack {Front) 027 Altix ICE Rack {Front) 021 Altix ICE Rack {Front) 024 Altix ICE Rack {Front) 022 Altix ICE Rack {Front) SWITCH Rack {Front) 023 Altix ICE Rack {Front) 018 Altix ICE Rack (Front) Archive Rack (Front) Storage Rack 001 (Front) Storage Rack 005 (Front) Storage Rack 004 (Front) Storage Rack 003 (Front) Storage Rack 002 Indicates overhead cable trellis Lustre IB connections Chilled Water & Condensate Encanto Floor Plan XE310 XE310 XE310 XE310 XE310 XE310 XE310 XE310 XE310 XE310

  14. Rack Leader Ctrl – 6015B Altix ICE 8200 Blade Enclosure (16 blades) Altix ICE Rack (Front) Altix ICE 8200 Blade Enclosure (16 blades) Altix ICE 8200 Blade Enclosure (16 blades) Altix ICE 8200 Blade Enclosure (16 blades) Encanto Compute Rack (28) Lustre OSS I/O Service Nodes (XE310 servers) are placed on top of the following ICE racks (5, 6, 7, 8, 9, 10, 11, 12, 24, 25)

  15. Altix 450 Blade Enclosure Altix 450 Blade Enclosure ADMIN Rack (Front) IS4000 Controller NAS CUBE & Archive IS4000 Drive Tray (16 drives) Administrator Console Sys Admin Ctrl – 6015B Batch Node – 6015B Login Node – 6015B Login Node – 6015B Login Node – 6015B Login Node – 6015B Login Node – 6015B Main Cluster Service Nodes Encanto Admin Rack IS4000 Drive Tray (16 drives)

  16. Cisco 6509 10-GigE Switch {Front) Support ICE Rack Administrator Console (Test) MDS Node – 6015B MDS Node – 6015B Lustre Meta Data Server IS220 (12 drives) Sys Admin Ctrl (Test) – 6015B Login Node (Test) – 6015B Test ICE System Altix ICE 8200 Blade Enclosure (Test) (16 blades) Encanto Switch Rack Rack Leader Ctrl (Test) – 6015B

  17. Lustre Storage Unit Lustre Storage Unit Storage Rack (Front) IS4500 Controller IS4500 Controller IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) IS4500 Drive Tray (16 drives) Encanto Luster Storage Rack (5)

  18. Archive & Backup Cartridge Storage Units Cartridge Expansion Unit STK SL500 (Front) Archive Rack Encanto Archive Library Cartridge Rack

  19. Exemplar System Overview • Major Components • 2.1 TFLOPS Altix ICE 8200 Cluster • 22 Nodes, 44 Sockets, 176 Cores • 352 TB memory (2GB/core) • 16 TB NAS Storage Subsystem • 1 System Admin Controller • 1 Rack Leader Controller • 1 Login Service Node • Lambda Rail Connectivity • 1 Consolidated Rack

  20. IB DDR Gig-E 4 10 Gig-E Exemplar Top-level Diagram Lamba Rail Fiber 1 System Admin Controller Lambda Rail Gateway Altix ICE 8200 Cluster 176 Compute Cores 22 Nodes 1 Rack Login Service Node 2 33 IB Lines 33 NAS Storage Subsystem 16 TB Raw

  21. FC FC SATA RAID Drive Tray SATA RAID Drive Tray Exemplar NAS Storage Subsystem 4 IB Altix 450 NAS Head Node FC FC 1 1 FC FC IS4000 RAID-controller

  22. Rack Leader Controller – 6015B NAS Storage CUBE Altix 450 IS4000 Controller and 16 Drives Drive Tray (16 Drives) Administrator Console Altix ICE 8200 Blade Enclosure (11 blades) Sys Admin Controller – 6015B Login Node – 6015B Altix ICE Cluster Altix ICE 8200 Blade Enclosure (11 blades) Exemplar System Rack

More Related