1 / 54

Overview of Parallel Architecture

Overview of Parallel Architecture Yulia Newton CS 147, Fall 2009 SJSU What is it? “Parallel processing is the ability of an entity to carry out multiple operations or tasks simultaneously. The term is used in the contexts of both human cognition and machine computation.” - Wikipedia Humans

Sophia
Download Presentation

Overview of Parallel Architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Overview of Parallel Architecture Yulia Newton CS 147, Fall 2009 SJSU

  2. What is it? • “Parallel processing is the ability of an entity to carry out multiple operations or tasks simultaneously. The term is used in the contexts of both human cognition and machine computation.” - Wikipedia

  3. Humans • Human body is a parallel processing architecture • Human body is the most complex machine known to man • We perform parallel processing all the time

  4. Why? • We want to do MORE • We want to do it FASTER

  5. Limitations of Single Processor • Physical • Heat • Electromagnetic interference • Economic • Cost

  6. Parallel Computing • Divide bigger tasks into smaller tasks • Reminds you Divide and Conquer concept? • Perform calculations of those tasks simultaneously (in parallel)

  7. With Single CPU • Parallel processing is possible with single-CPU, single-core computers • Requires very sophisticated software called distributed processing software • Most parallel computing is done with multiple CPU’s

  8. Speedup from parallelization • More processors = faster computing? • Speed-up from parallelization is not linear Can do single task in a unit of time Can do twice as many tasks in same the amount of time Can do four times as many tasks in the same amount of time

  9. Speedup • Definition: Speedup is how much a parallel algorithm is faster than a corresponding sequential algorithm.

  10. Gene Myron Amdahl • Norwegian American computer architect and hi-tech entrepreneur

  11. Amdahl's Law • Formulated in 1960s • Gives potential speed-up on parallel computing system Small portion of components/elements that cannot be parallelized will limit speed-up possible from parallelizable components/elements.

  12. Amdahl's Law (cont’d) where : • S is the overall speed-up of the system • P is the fraction that is parallelizable

  13. Amdahl's Law Alternative Form • where • f – fraction of the program that is enhanced • s – speedup of the enhanced portion

  14. Amdahl's Law (cont’d)

  15. Amdahl's Law Example Program • Program has four parts Code 1 Part 1 = 11% Loop 1 Part 2 = 18% Loop 2 Part 3 = 23% Code 2 Part 4 = 48%

  16. Amdahl's Law Example (cont’d) • Let’s say: • P1 is not sped up, so S1 = 1 or 100% • P2 is sped up 5×, so S2 = 500% • P3 is sped up 20×, so S3 = 2000% • P4 is sped up 1.6×, so S4 = 160%

  17. Amdahl's Law Example (cont’d) • Using P1/S1 + P2/S2 + P3/S3 + P4/S4:

  18. Amdahl's Law Example (cont’d) • 0.4575 is a little less than ½ of the original running time, which is 1 • Overall speed boost is 1 / 0.4575 = 2.186 or a little more than double the original speed • Notice how the 20× and 5× speedup don't have much effect on the overall speed boost and running time when 11% is not sped up, and 48% is sped up by 1.6×

  19. Amdahl's Law Limitations • Based on fixed work load or fixed problem size (strong or hard scaling) • Implies that machine size is irrelevant

  20. John L. Gustafson • American computer scientist and businessman

  21. Gustafson's Law • Formulated in 1988 • Closely related to Amdahl’s Law • Addresses the shortcomings of Amdahl's law, which cannot scale to match availability of computing power as the machine size increases

  22. Gustafson's Law (cont’d) • where P is the number of processors, S is the speedup, and α the non-parallelizable part of the process

  23. Gustafson's Law (cont’d) • Proposes a fixed time concept which leads to scaled speed up for larger problem sizes • Uses weak or soft scaling

  24. Gustafson's Law Limitations • Some problems do not have fundamentally larger datasets • Example: processing one data point per world citizen gets larger at only a few percent per year • Nonlinear algorithms may make it hard to take advantage of parallelism "exposed" by Gustafson's law

  25. Parallel Architectures • Superscalar • VLIW • Vector Processors • Interconnection Networks • Shared Memory Multiprocessors • Distributed Computing • Dataflow Computing • Neural Networks • Systolic Arrays

  26. Superscalar Architecture • Implements instruction-level parallelism (multiple instructions are executed simultaneously in each cycle) • Net effect is the same as pipelining • Additional hardware is required • Contains specialized instruction fetch unit (retrieves multiple instructions simultaneously from memory) • Relies on both hardware and compiler

  27. Superscalar Architecture • Analogous to adding a lane to a highway – more physical resources are required but in the end more cars can pass through

  28. Superscalar Architecture • Examples: • Pentium x86 processors • Intel i960CA • AMD 29000-series

  29. VLIW Architecture • Stands for Very Long Instruction Word • Similar to Superscalar architecture • Relies entirely on compiler • Pack independent instructions into one long instruction • Bigger size of compiled code

  30. VLIW Architecture • Examples: • Intel’s Itanium, IA-64 • Floating Point Systems' FPS164

  31. Vector Processors • Supercomputers are vector processor systems • Specialized, heavily pipelined processors • Efficient operations on entire vectors and matrices

  32. Vector Processors • Heavily pipelined so that arithmetic operations can be overlapped

  33. Vector Processors (example) • Instructions on a traditional processor: For I = 0 to VectorLength V3[i] = V1[i] + V2[i] • Instructions on a vector processor: LDV V1, R1 LDV V2, R2 ADDV R3, R1, R2 STV R3, V3

  34. Vector Processors • Applications: • Weather forecasting • Medical diagnoses systems • Image processing

  35. Vector Processors • Examples: • Cray series of cupercomputers • Xtrillion 3.0

  36. Interconnection Networks • MIMD (Multiple instruction stream, multiple data stream) type architecture • Each processor has its own memory • Processors are allowed to access other processors’ memory via network

  37. Interconnection Networks • Can be: • Dynamic – allow paths between two components to change from one communication to another • Static – do not allow a change of path between communications

  38. Interconnection Networks • Can be: • Non-blocking – allow new connections in the presence of another simultaneous connections • Blocking – do not allow simultaneous connections

  39. Interconnection Networks • Sample interconnection networks:

  40. Shared Memory Architecture • MIMD (Multiple instruction stream, multiple data stream) type architecture • Single memory pool accessed by all processors • Can be categorized by how the memory access is performed: • UMA (Uniform Memory Access) – all processors have equal access to entire memory • NUMA (Non-Uniform Memory Access) – each processor has access to its own piece of memory

  41. Shared Memory Architecture • Diagram of a Shared Memory system:

  42. Distributed Computing • Very loosely coupled multicomputer system, connected by buses or through a network • Main advantage is cost • Main disadvantage is slow communication due to the physical distance between computing units

  43. Distributed Computing • Example: • SETI (Search for Extra Terrestrial Intelligence) group at UC Berkley analyzes data from radio-telescopes • To help this project PC users can install SETI screen saver on their home computer • Data will be analyzed during the processor’s idle time • Half a million years of CPU time accumulated in about 18 months!

  44. Data Flow Computing • Von Neumann machines exhibit sequential control flow; data and instructions are segregated • Dataflow computing – control of the program is directly tied to the data • Order of the instructions does not matter • Each instruction is considered a separate process • Tokens are executed rather than instructions

  45. Data Flow Computing • Data Flow Graph for y=(c+3)*f

  46. Data Flow Computing • Data flow system diagram:

  47. Neural Networks • Good for massively parallel applications, fault tolerance and adapting to changing circumstances • Based on parallel architecture of human brain • Composed of large number of simple processing elements • Trainable

  48. Neural Networks • Model human brain • Perceptron is the simplest example

  49. Neural Networks • Applications: • Quality control • Financial and economic forecasting • Oil and gas exploration • Speech and pattern recognition • Health care cost reduction • Bankruptcy prediction

  50. Systolic Arrays • Networks of processing elements that rhythmically compute data by circulating it through the system • Variations of SIMD architecture

More Related