1 / 77

Parallel computing

Parallel computing. Reference https://computing.llnl.gov/tutorials/parallel_comp/ Parallel Programming in C with MPI and OpenMP. 平行計算 賴寶蓮. TextBook. Introduction to Parallel Computing George Karypis, Vipin Kumar 2003, Pearson 全華代理. 參考教材.

hazina
Download Presentation

Parallel computing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Parallel computing Reference https://computing.llnl.gov/tutorials/parallel_comp/ Parallel Programming in C with MPI and OpenMP 平行計算 賴寶蓮

  2. TextBook • Introduction to Parallel Computing • George Karypis, Vipin Kumar • 2003, Pearson • 全華代理

  3. 參考教材 • Parallel Programming in C with MPI and OpenMP, McGraw Hill,2004 • Patterns for Parallel Programming, Addison-Wesley,2004 • Parallel Programming with MPI, Peter S. Pacheco, Morgan Kaufmann Publishers, 1997 • OpenMP Specification, www.openmp.org • www.top500.org • Parallel Computing • International Journal of Parallel Programming

  4. von Neumann Architecture Comprised of four main components: Memory Control Unit Arithmetic Logic Unit Input/Output

  5. Serialcomputing • Traditionally, software has been written for serial computation: • To be run on a single computer having a single Central Processing Unit (CPU); • A problem is broken into a discrete series of instructions. • Instructions are executed one after another. • Only one instruction may execute at any moment in time.

  6. Parallel computing • In the simplest sense, parallel computing is the simultaneous use of multiple compute resources to solve a computational problem: • To be run using multiple CPUs • A problem is broken into discrete parts that can be solved concurrently • Each part is further broken down to a series of instructions • Instructions from each part execute simultaneously on different CPUs

  7. Parallel computing • The compute resources can include: • A single computer with multiple processors; • An arbitrary number of computers connected by a network; • A combination of both.

  8. Parallel computing • The computational problem usually demonstrates characteristics such as the ability to be: • Broken apart into discrete pieces of work that can be solved simultaneously; • Execute multiple program instructions at any moment in time; • Solved in less time with multiple compute resources than with a single compute resource.

  9. The Universe is Parallel: • Parallel computing is an evolution of serial computing that attempts to emulate what has always been the state of affairs in the natural world: • many complex, interrelated events happening at the same time, yet within a sequence. • For example: • Galaxy銀河formation • Planetary行星的movement • Weather and ocean patterns • Tectonic築造的plate drift • Rush hour traffic • Automobile assembly line • Building a space shuttle • Ordering a hamburger at the drive through.

  10. Uses for Parallel Computing: • Historically, parallel computing has been considered to be "the high end of computing", and has been used to model difficult scientific and engineering problems found in the real world. • Some examples: • Atmosphere, Earth, Environment • Physics - applied, nuclear, particle, condensed matter, high pressure, fusion, photonics • Bioscience, Biotechnology, Genetics • Chemistry, Molecular Sciences • Geology, Seismology • Mechanical Engineering - from prosthetics to spacecraft • Electrical Engineering, Circuit Design, Microelectronics • Computer Science, Mathematics

  11. Uses for Parallel Computing: • Today, commercial applications provide an equal or greater driving force in the development of faster computers. These applications require the processing of large amounts of data in sophisticated ways. • For example: • Databases, data mining • Oil exploration • Web search engines, web based business services • Medical imaging and diagnosis • Pharmaceutical design • Management of national and multi-national corporations • Financial and economic modeling • Advanced graphics and virtual reality, particularly in the entertainment industry • Networked video and multi-media technologies • Collaborative work environments

  12. Why Use Parallel Computing? • Main Reasons: • Save time and/or money: • Solve larger problems: • Provide concurrency: • Use of non-local resources: • Limits to serial computing:

  13. Why Use Parallel Computing? • Save time and/or money: • In theory, throwing more resources at a task will shorten its time to completion, with potential cost savings. • Parallel clusters can be built from cheap, commodity components.

  14. Why Use Parallel Computing? • Solve larger problems: • Many problems are so large and/or complex that it is impractical or impossible to solve them on a single computer, especially given limited computer memory. • For example: • "Grand Challenge" (en.wikipedia.org/wiki/Grand_Challenge) problems requiring PetaFLOPS and PetaBytes of computing resources. • Web search engines/databases processing millions of transactions per second

  15. Why Use Parallel Computing? • Provide concurrency: • A single compute resource can only do one thing at a time. • Multiple computing resources can be doing many things simultaneously. • For example, the Access Grid (www.accessgrid.org) provides a global collaboration network where people from around the world can meet and conduct work "virtually".

  16. Why Use Parallel Computing? • Use of non-local resources: • Using compute resources on a wide area network, or even the Internet when local compute resources are scarce. • For example: • SETI@home (setiathome.berkeley.edu) uses over 330,000 computers for a compute power over 528 TeraFLOPS (as of August 04, 2008) • Folding@home (folding.stanford.edu) uses over 340,000 computers for a compute power of 4.2 PetaFLOPS (as of November 4, 2008) One petaflops is equal to 1,000 teraflops, or 1,000,000,000,000,000 FLOPS.

  17. Why Use Parallel Computing? • Limits to serial computing: • Both physical and practical reasons pose significant constraints to simply building ever faster serial computers: • Transmission speeds • Limits to miniaturization小型化 • Economic limitations

  18. Why Use Parallel Computing? • Current computer architectures are increasingly relying upon hardware level parallelism to improve performance: • Multiple execution units • Pipelined instructions • Multi-core

  19. Who and What ? • Top500.org provides statistics on parallel computing users – • the charts below are just a sample. • Some things to note: Sectors may overlap – • for example, research may be classified research. Respondents have to choose between the two. • "Not Specified" is by far the largest application - probably means multiple applications.

  20. The Future: • During the past 20 years, the trends indicated by ever faster networks, distributed systems, and multi-processor computer architectures (even at the desktop level) clearly show that parallelism is the future of computing.

  21. Modern Parallel Computers • Caltech’s Cosmic Cube (Seitz and Fox) • Commercial copy-cats • nCUBE Corporation • Intel’s Supercomputer Systems Division • Lots more • Thinking Machines Corporation

  22. Modern Parallel Computers • Cray Jaguar (224162 cores, 1.75PFlops) • IBM Roadrunner (122400 cores, 1.04PFlops) • Cray Kraken XT5 (98928 cores, 831TFlops) • IBM JUGENE (294912 cores, 825TFlops) • NUDT Tianhe-1 (71680 cores, 563TFlops) • (2009/11 TOP 5) • IBM 1350 (台灣國網中心) (2048 cores, 23TFlops)

  23. Seeking Concurrency • Data dependence graphs • Data parallelism • Functional parallelism • Pipelining

  24. Data Dependence Graph • Directed graph • Vertices = tasks • Edges = dependences

  25. Data Parallelism • Independent tasks apply same operation to different elements of a data set • Okay to perform operations concurrently for i  0 to 99 do a[i]  b[i] + c[i] endfor

  26. Functional Parallelism • Independent tasks apply different operations to different data elements • First and second statements • Third and fourth statements a  2 b  3 m  (a + b) / 2 s  (a2 + b2) / 2 v  s - m2

  27. Pipelining • Divide a process into stages • Produce several items simultaneously

  28. Partial Sums Pipeline

  29. Data Clustering • Data mining • looking for meaningful patterns in large data sets • Data clustering • organizing a data set into clusters of “similar” items • Data clustering can speed retrieval of related items

  30. A graphical representation of Amdahl's law. • The speedup of a program using multiple processors in parallel computing is limited by the sequential fraction of the program. • For example, • if 95% of the program can be parallelized, the theoretical maximum speedup using parallel computing would be 20× as shown in the diagram, no matter how many processors are used.

  31. Programming Parallel Computers • Extend compilers: • translate sequential programs into parallel programs • Extend languages: • add parallel operations • Add parallel language layer on top of sequential language • Define totally new parallel language and compiler system

  32. Strategy 1: Extend Compilers • Parallelizing compiler • Detect parallelism in sequential program • Produce parallel executable program • Focus on making Fortran programs parallel

  33. Extend Compilers (cont.) • Advantages • Can leverage millions of lines of existing serial programs • Saves time and labor • Requires no retraining of programmers • Sequential programming easier than parallel programming

  34. Extend Compilers (cont.) • Disadvantages • Parallelism may be irretrivably lost when programs written in sequential languages • Performance of parallelizing compilers on broad range of applications still up in air

  35. Extend Language • Add functions to a sequential language • Create and terminate processes • Synchronize processes • Allow processes to communicate

  36. Extend Language (cont.) • Advantages • Easiest, quickest, and least expensive • Allows existing compiler technology to be leveraged • New libraries can be ready soon after new parallel computers are available

  37. Extend Language (cont.) • Disadvantages • Lack of compiler support to catch errors • Easy to write programs that are difficult to debug

  38. Current Status • Low-level approach is most popular • Augment existing language with low-level parallel constructs (by function call) • MPI and OpenMP are examples • Advantages of low-level approach • Efficiency • Portability • Disadvantage: More difficult to program and debug

  39. OpenMP • 使用Shared Memory的架構 https://computing.llnl.gov/tutorials/parallel_comp/

  40. MPI • MPI是一種standard • 使用Distributed Memory • 市面上比較流行的軟體有LAM和MPICH。 https://computing.llnl.gov/tutorials/parallel_comp/

  41. Organization and Contents of this Course • Fundamentals: This part of the class covers • basic parallel platforms • principles of algorithm design • group communication primitives, and • analytical modeling techniques.

  42. Organization and Contents of this Course • Parallel Programming: This part of the class deals with programming using • message passing libraries and • threads. • Parallel Algorithms: This part of the class covers basic algorithms for • matrix computations, • graphs, sorting, • discrete optimization, and • dynamic programming.

  43. 課程大綱 • Introduction to Parallel Computing • Parallel Programming Platforms • Principles of Parallel Algorithm Design • Basic Communication Operations • Analytical Modeling of Parallel Programs • Programming Using the Message-Passing Paradigm (MPI) • 期中報告(or 期中考試)

  44. 課程大綱 • Programming Shared Address Space Platforms (OpenMP) • Dense Matrix Algorithms • Sorting Algorithms • Graph Algorithms • Hadoop 簡介與安裝架設 • Map-Reduce 程式架構簡介 • Map-Reduce 範例程式介紹與實作 • 期末計劃 (or期末報告)

  45. 計分 • 平時成績(上課,出席,實作,作業)40% • 期中報告(or 期中考試) 30% • 期末計劃(or期末報告) 30%

  46. P2P system • Peer-to-peer system • Client acts as a server • Share data • BT

  47. http://www.fidis.net/typo3temp/tx_rlmpofficelib_0c97e8a6cd.pnghttp://www.fidis.net/typo3temp/tx_rlmpofficelib_0c97e8a6cd.png

  48. Grid Computing • Distributed computing • Ethernet/ Internet • Volunteer computing networks • Software-as-a-service (SaaS) • Software that is owned, delivered and managed remotely by one or more providers.

  49. http://www.csa.com/discoveryguides/grid/images/gridcomp.gif

  50. Cloud Computing • 雲端運算不是技術，它是概念 • Distributed computing • Web services

More Related