Distributed Parallel Computing

1. Distributed Parallel Computing R&D rnd@ciit.net.pk http://202.179.135.4/rnd

2. Outline • Introduction • Why we need powerful computers • Why powerful computers are parallel • Parallel computers, yesterday and today • Issues in parallel performance • What are Clusters • How do I write parallel apps • CIIT Computation resources • Aplications Areas • QA

3. Introduction • R&D resource managers • Usman • Imtiaz rnd@ciit.net.pk • Course Instructor @ CIIT : • Kashif Bilal kashifbilal@ciit.net.pk

4. Why do we need powerful computers?

5. Simulation: The Third Pillar of Science • Traditional scientific and engineering paradigm: • Do theory or paper design. • Perform experiments or build system. • Limitations: • Too difficult -- build large wind tunnels. • Too expensive -- build a throw-away passenger jet. • Too slow -- wait for climate or galactic evolution. • Too dangerous -- weapons, drug design, climate experiments. • Computational science paradigm: • Use high performance computer systems to simulate the phenomenon. • Base on known physical laws and efficient numerical methods.

6. Some Challenging Computations • Science • Global climate modeling • Astrophysical modeling • Biology: genomics; protein folding; drug design • Computational Chemistry • Computational Material Sciences and Nanosciences • Engineering • Crash simulation • Semiconductor design • Earthquake and structural modeling • Computation fluid dynamics (airplane design) • Combustion (engine design) • Business • Financial and economic modeling • Transaction processing, web services and search engines • Defense • Nuclear weapons -- test by simulation • Cryptography

7. Units of Measure in HPC • High Performance Computing (HPC) units are: • Flop/s: floating point operations per second • Typical sizes are millions, billions, trillions… Mega Mflop/s = 106 flop/sec Mbyte = 106 byte Giga Gflop/s = 109 flop/sec Gbyte = 109 byte Tera Tflop/s = 1012 flop/sec Tbyte = 1012 byte Peta Pflop/s = 1015 flop/sec Pbyte = 1015 byte Exa Eflop/s = 1018 flop/sec Ebyte = 1018 byte

8. Global Climate Modeling Problem • Problem is to compute: f(latitude, longitude, elevation, time)  temperature, pressure, humidity, wind velocity • Approach: • Discretize the domain, e.g., a measurement point every 10 km • Devise an algorithm to predict weather at time t+1 given t • Uses: • Predict major events, e.g., El Nino • Use in setting air emissions standards Source: http://www.epm.ornl.gov/chammp/chammp.html

9. Global Climate Modeling Computation • One piece is modeling the fluid flow in the atmosphere • Solve Navier-Stokes problem • Roughly 100 Flops per grid point with 1 minute timestep • Computational requirements: • To match real-time, need 5x 1011 flops in 60 seconds = 8 Gflop/s • Weather prediction (7 days in 24 hours)  56 Gflop/s • Climate prediction (50 years in 30 days)  4.8 Tflop/s • To use in policy negotiations (50 years in 12 hours)  288 Tflop/s • To double the grid resolution, computation is at least 8x • State of the art models require integration of atmosphere, ocean, sea-ice, land models, plus possibly carbon cycle, geochemistry and more • Current models are coarser than this http://www.nersc.gov/aboutnersc/pubs/bigsplash.pdf

10. Why are powerful computers parallel?

11. Tunnel Vision by Experts • “I think there is a world market for maybe five computers.” • Thomas Watson, chairman of IBM, 1943. • “There is no reason for any individual to have a computer in their home” • Ken Olson, president and founder of Digital Equipment Corporation, 1977. • “640K [of memory] ought to be enough for anybody.” • Bill Gates, chairman of Microsoft,1981. Slide source: Warfield et al.

12. Technology Trends: Microprocessor Capacity Moore’s Law Moore’s Law: #transistors/chip doubles every 1.5 years Gordon Moore (co-founder of Intel) predicted in 1965 that the transistor density of semiconductor chips would double roughly every 18 months. Microprocessors have become smaller, denser, and more powerful. Slide source: Jack Dongarra

13. How fast can a serial computer be? • Consider the 1 Tflop sequential machine • data must travel some distance, r, to get from memory to CPU • to get 1 data element per cycle, this means 10^12 times per second at the speed of light, c = 3e8 m/s • so r < c/10^12 = .3 mm • Now put 1 TB of storage in a .3 mm^2 area • each word occupies ~ 3 Angstroms^2, the size of a small atom • (1 Angstrom = 0.000,000,1 mm) 1 Tflop 1 TB sequential machine r = .3 mm

14. “Automatic” Parallelism in Modern Machines • Bit level parallelism • within floating point operations, etc. • Instruction level parallelism • multiple instructions execute per clock cycle • Memory system parallelism • overlap of memory operations with computation • OS parallelism • multiple jobs run in parallel on commodity SMPs There are limits to all of these -- for very high performance, user must identify, schedule and coordinate parallel tasks

15. Number of transistors per processor chip

16. Number of transistors per processor chip Instruction-Level Parallelism Thread-Level Parallelism? Bit-Level Parallelism

17. Parallel computers, yesterday and today

18. Various Competing Computer Architectures • Vector Computers (VC) ---proprietary system • provided the breakthrough needed for the emergence of computational science, buy they were only a partial answer. • Massively Parallel Processors (MPP)-proprietary system • high cost and a low performance/price ratio. • Symmetric Multiprocessors (SMP) • suffers from scalability • Clusters -- gaining popularity • High Performance Computing---Commodity Supercomputing • High Availability Computing ---Mission Critical Applications

19. Memory Memory Memory Memory CPU CPU CPU CPU ..Introduction of PDC Models • Shared Memory • Distributed Memory

20. Machine ArchitecturesShared Memory CPU1 CPU2 CPUN - - - - - - - - - - NETWORK MEMORY FEATURES: 1) All CPUs share memory 2) CPUs access memory using the interconnection network Example: Top Gun, Kirk, every node of the Itanium2 cluster is a dual-processor SMP machine

21. CPU2 CPU1 CPU4 CPU3 Local Memory Local Memory Local Memory Local Memory Machine ArchitecturesDistributed Memory Network FEATURES: 1) Each node has its own local memory 2) Nodes share data by passing data over the network Example: Sun Solaris/intel Workstations

22. Issues in parallel performance

23. Locality and Parallelism Conventional Storage Hierarchy Proc Proc Proc • Large memories are slow, fast memories are small • Storage hierarchies are large and fast on average • Parallel processors, collectively, have large, fast cache • the slow accesses to “remote” data we call “communication” • Algorithm should do most work on local data Cache Cache Cache L2 Cache L2 Cache L2 Cache L3 Cache L3 Cache L3 Cache potential interconnects Memory Memory Memory

24. Finding Enough Parallelism: Amdahl’s Law Amdahl’s Law How many processors can we really use? Let’s say we have a legacy code such that is it only feasible to convert half of the heavily used routines to parallel:

25. Finding Enough Parallelism: Amdahl’s Law Amdahl’s Law • If we run this on a parallel machine with five processors: • Our code now takes about 60s. We have sped it up by about 40%. Let’s say we use a thousand processors: • We have now sped our code by about a factor of two.

26. Finding Enough Parallelism: Amdahl’s Law • Suppose only part of an application seems parallel • Amdahl’s law • Let s be the fraction of work done sequentially, so (1-s) is the fraction parallelizable • Let P = number of processors Speedup(P) = Time(1)/Time(P) <= 1/(s + (1-s)/P) <= 1/s • Even if the parallel part speeds up perfectly, the sequential part limits overall performance.

27. Finding Enough Parallelism: Amdahl’s Law Amdahl’s Law speedup and efficiency : • Speedup = SN = Ts / Tp • Efficiency = EN = SN / N • If the best known serial algorithm takes 8 seconds i.e. Ts = 8, while a parallel algorithm takes 2 seconds using 5 processors, then • SN = Ts / Tp = 8 / 2 = 4 and • EN = SN / N = 4 / 5 = 0.8 = 80% • i.e. the parallel algorithm exhibits a speedup of 4 with 5 processors giving an 80% efficiency.

28. Load Imbalance • Load imbalance is the time that some processors in the system are idle due to • insufficient parallelism (during that phase) • unequal size tasks • Examples of the latter • adapting to “interesting parts of a domain” • tree-structured computations • fundamentally unstructured problems • Algorithm needs to balance the load

29. Some of the Fastest Super Computers

30. Parallel Computing Today IBM BlueGene @ 280 TFlops

31. Parallel Computing Today Mini BlueGene @ ~91 TFlops

32. Parallel Computing Today ASC Purple @ 63 TFlops

33. Parallel Computing Today Columbia SGI Altix @ 51 TFlops

34. Parallel Computing Today Earth Simulator @ 35 TFlops

35. Parallel Computing Today

36. Parallel Computing @ home Small class Beowulf cluster

37. Clusters

38. What is a cluster? • A cluster is a type of parallel or distributed processing system, which consists of a collection of interconnected stand-alone computers cooperatively working together as a single, integrated computing resource. • A typical cluster: • Network: Faster, closer connection than a typical network (LAN) • Low latency communication protocols • Looser connection than SMP

39. Cluster Architecture

40. Backbone/Communication Topology

41. Token-Ring/Ethernet with Workstations

42. Complete Connectivity

43. Star Topology

44. Binary Tree

45. INTEL Paragon (2-D Mesh)

46. The Need for Alternative Supercomputing Resources • Cannot afford to buy “Big Iron” machines • due to their high cost and short life span. • cut-down of funding • don’t “fit” better into today's funding model. • …. • Paradox: time required to develop a parallel application for solving GCA is equal to: • half Life of Parallel Supercomputers.

47. Clusters are best-alternative! • Supercomputing-class commodity components are available • They “fit” very well with today’s/future funding model. • Can leverage upon future technological advances • VLSI, CPUs, Networks, Disk, Memory, Cache, OS, programming tools, applications,...

48. Best of both Worlds! • High Performance Computing • parallel computers/supercomputer-class workstation cluster • dependable parallel computers • High Availability Computing • mission-critical systems • fault-tolerant computing

49. So What’s So Different about Clusters? • Commodity Parts? • Communications Packaging? • Incremental Scalability? • Independent Failure? • Intelligent Network Interfaces? • Complete System on every node • virtual memory • scheduler • files • … • Nodes can be used individually or combined...

50. 1984 Computer Food Chain Mainframe PC Workstation Mini Computer Vector Supercomputer