Future of High Performance Computing

1. Future of High Performance Computing Thom Dunning National Center for Supercomputing Applications

2. Outline of Presentation • Directions in Computing Technology • From uni-core to multi-core chips • On to many-core chips • From Terascale to Petascale Computing • Science @ Petascale • Blue Waters Petascale Computing System • Path to Exascale Computing • Issues for beyond petascale computing • Take Home Lessons Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

3. Directions in Computing Technology A major shift is underway in computing technology with multicore and many-core chips

4. Directions in Computing TechnologyIncreasing Performance of Microprocessors “In the past, performance scaling in conventional single-core processors has been accomplished largely through increases in clock frequency (accounting forroughly 80 percent of the performance gains to date).” Platform 2015 S. Y. Borkar et al., 2006 Intel Corporation Frequency (MHz) Intel Pentium Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

5. Directions in Computing TechnologyProblem with Uni-core Microprocessors Rocket Nozzle 1000 Nuclear Reactor Pentium 4 (Prescott) 100 Pentium 4 (Willamette) Watts/cm2 Pentium III Hot Plate Pentium II 10 Pentium Pro Pentium i386 i486 1 1.5m 1.0m 0.7m 0.5m 0.35m 0.25m 0.18m 0.13m 0.1m 0.07m Decreasing Feature Size Increasing Chip Frequency Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

6. Directions in Computing TechnologyFrom Uni-core to Multi-core Processors Intel’s Nehalem • Modular • Up to 8 cores • 3 levels of cache • Integrated memory controller • Multiple QuickPath Interconnects Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

7. Directions in Computing TechnologySwitch to Multicore Chips “For the next several years the only way to obtain significant increases in microprocessor performance will be through increasing use of parallelism: – 8× in 2009-10, – 16× in 2011-12, – and so on dual core quad core Frequency (MHz) Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

8. Directions in Computing TechnologyOn to Many-core Chips AMD Llano (4 x86 cores + 480 stream processors) NVIDIA Fermi (512 cores) Intel Teraflops Chip (80 cores) Intel Many Integrated Cores (>80 x86+ cores) Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

9. Directions in Computing TechnologyRecent Evolution of NVIDIA GPUs Peak DP performance = 256 FMA/cycle x 2 flops/FMA x 1.5 GHz = 768 GF Peak SP performance = 512 FMA/cycle x 2 flops/FMA x 1.5 GHz = 1,536 GF Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

10. Directions in Computing TechnologyFermi Streaming Multiprocessor Architecture • Streaming Multiprocessor • 16 SMs per chip • Each SM has: • 32 CUDA cores • Floating point and integer units for each core • Fused multiply-add instruction • 16 load-store units • 4 special function units • Transcendental functions (sin, cos, reciprocal, square root) Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

11. Directions in Computing TechnologyAMD’s Fusion “Application Processing Unit” • Heterogeneous Architecture • x86 cores • Streaming processors • High Performance Interconnect • High Performance Memory Controller Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

12. Blue Waters: From Terascale to Petascale Computing A computing system for solving the most challenging compute-, memory- and data-intensive problems

13. Blue WatersNSF Track 1 Solicitation “The petascale HPC environment will enable investigations of computationally challenging problems that require computing systems capable of delivering sustained performance approaching 1015 floating point operations per second (petaflops) on real applications, that consume large amounts of memory, and/or that work with very large data sets.” Leadership-Class System Acquisition - Creating a Petascale Computing Environment for Science and Engineering NSF 06-573 Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

14. Blue WatersComputational Science and Engineering Petascale computing will enable advances in a broad range of science and engineering disciplines: Molecular Science Weather & Climate Forecasting Health Astronomy Earth Science Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

15. Blue WatersDesired Attributes of Petascale System • Maximum Core Performance … to minimize number of cores needed for a given performance level, lessen impact of sections of code with limited scalability • Low Latency, High Bandwidth Interconnect … to enable science and engineering applications to scale to tens to hundreds of thousands of cores • Large, Fast Memories … to solve the most memory-intensive problems • Large, Fast I/O System and Data Archive … to solve the most data-intensive problems • Reliable Operation … to enable the solution of Grand Challenge problems Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

16. Blue WatersBuilding Blue Waters Blue Waters will be the most powerful computer in the world for scientific research when it comes on line in Summer of 2011. Blue Waters ~10 PF Peak ~1 PF sustained >300,000 cores ~1.2 PB of memory >18 PB of disk storage 500 PB of archival storage ≥100 Gbps connectivity Blue Waters Building Block 32 IH server nodes 256 TF (peak) 32 TB memory 128 TB/s memory bw 4 Storage systems (>500 TB) 10 Tape drive connections IH Server Node 8 QCM’s (256 cores) 8 TF (peak) 1 TB memory 4 TB/s memory bw 8 Hub chips Power supplies PCIe slots Fully water cooled Quad-chip Module 4 Power7 chips 128 GB memory 512 GB/s memory bw 1 TF (peak) Hub Chip 1,128 GB/s bw Blue Waters is built from components that can be used to build systems with a wide range of capabilities—from servers to beyond Blue Waters. Power7 Chip 8 cores, 32 threads L1, L2, L3 cache (32 MB) Up to 256 GF (peak) 128 Gb/s memory bw 45 nm technology Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

17. Blue WatersComparison: Jaguar and Blue Waters ORNL NCSA System Attribute Jaguar (#1) Blue Waters Vendor (Model) Cray (XT5) IBM (PERCS) Processor AMD Opteron IBM Power7 Peak Performance (PF) 2.3 ~10 Sustained Performance (PF) ? ≳1 Number of Cores/Chip 6 8 Number of Processor Cores 224,256 >300,000 Amount of Memory (TB) 299 ~1,200 Amount of On-line Disk Storage (PB) 5 >18 Sustained Disk Transfer (TB/sec) 0.24 >1.5 Amount of Archival Storage (PB) 20 up to 500 ~4 1⅓ <1½ 4 >3 >6 25 Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

18. Blue Waters ProjectCritical Features of Blue Waters. I • High Performance Compute Module • SMP system • Four Power7 chips • Hub chip • Performance: 1 TF • Memory: 128 GB • High Performance Interconnect • High bandwidth, low latency • Hub Chip/QCM: >1 TB/sec/QCM • Latency: ~1 msec • Fully connected, two tier network • Copper + optical links Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

19. Blue Waters ProjectCritical Features of Blue Waters. II • High Performance I/O and Data archive Systems • Large storage subsystems • On-line disks: >18 PB (usable) • Archival tapes: Up to 500 PB • High sustained disk transfer rate: >1.5 TB/sec (sustained) • Fully integrated storage system: GPFS + HPSS • General • Hardware support for global shared memory Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

20. Blue WatersNational Petascale Computing Facility Partners EYP MCF/ Gensler IBM Yahoo! • Energy Efficiency • LEED certified Gold (goal: Platinum) • PUE = 1.1–1.2 • Modern Data Center • 90,000+ ft2 total • 30,000 ft2 raised floor • 20,000 ft2 machine gallery Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

21. Path to Exascale Computing Although an exascale computer is at least 10 years away, the issues being confronted will impact all systems beyond Blue Waters Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

22. Blue WatersA Glimpse into the Future: Sequoia NCSA LLNL System Attribute Blue Waters Sequoia Vendor (Model) IBM (PERCS) IBM BG/Q Processor IBM Power7 IBM PowerPC Peak Performance (PF) ~10 ~20 Sustained Performance (PF) ≳1 ? Number of Cores/Chip 8 16 Number of Processor Cores >300,000 ~1,600,000 Amount of Memory (TB) ~1,200 ~1,600 Amount of On-line Disk Storage (PB) >18 ~50 Sustained Disk Transfer (TB/sec) >1.5 0.5–1.0 Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

23. Path from Petascale to Exascale • Levels of concurrency • Cores: 100s of thousands ➙100s of millions • Threads: million ➙ billion • Clock Rate of Core • No significant increase • Memory per Core • 1-4 GB ➙ 10s–100s of MB • Aggressive Fault Management in HW and SW • Power Consumption • 10 MW ➙ 40 MW – 150 MW Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

24. Take Home Lessons • Examine New Computing Technologies • Computers of future will be based on many-core chips • Details TBD, but may be heterogeneous • Focus on Scalable Algorithms • Only significant speed gains in future will come through increased parallelization • Explore New Programming Models • Computing systems will be (are!) collections of SMPs • Need to assess and improve MPI/OpenMP, UPC, CAF • Enhance Reliability • Systems level (e.g., virtualization) • Applications level Petascale Summer School • 6-9 July 2010 • Urbana, Illinois

25. Questions? Private Sector Program Annual Meeting • 12-14 May 2008 • Urbana, Illinois