1 / 32

Shared Memory and Message Passing

Shared Memory and Message Passing. W+A 1.2.1, 1.2.2, 1.2.3, 2.1, 2.1.2, 2.1.3, 8.3, 9.2.1, 9.2.2 Akl 2.5.2. Models for Communication. Parallel program = program composed of tasks (processes) which communicate to accomplish an overall computational goal

livi
Download Presentation

Shared Memory and Message Passing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Shared Memory and Message Passing W+A 1.2.1, 1.2.2, 1.2.3, 2.1, 2.1.2, 2.1.3, 8.3, 9.2.1, 9.2.2 Akl 2.5.2 CSE 160/Berman

  2. Models for Communication • Parallel program = program composed of tasks (processes) which communicate to accomplish an overall computational goal • Two prevalent models for communication: • Message passing (MP) • Shared memory (SM) • This lecture will focus on MP and SM computing CSE 160/Berman

  3. Message Passing Communication • Processes in message passing program communicate by passing messages • Basic message passing primitives • Send(parameter list) • Receive(parameter list) • Parameters depend on the software and can be complex A B CSE 160/Berman

  4. Flavors of message passing • Synchronous used for routines that return when the message transfer has been completed • Synchronous send waits until the complete message can be accepted by the receiving process before sending the message (send suspends until receive) • Synchronous receive will wait until the message it is expecting arrives (receive suspends until message sent) • Also called blocking request to send A B acknowledgement message CSE 160/Berman

  5. message buffer A B Nonblocking message passing • Nonblocking sends return whether or not the message has been received • If receiving processor not ready, message may be stored in message buffer • Message buffer used to hold messages being sent by A prior to being accepted by receive in B • MPI: • routines that use a message buffer and return after their local actions complete are blocking (even though message transfer may not be complete) • Routines that return immediately are non-blocking CSE 160/Berman

  6. InterconnectionNetwork Processor … Localmemory Basic Message Passing Multicomputer Architectural support for MP • Interconnection network should provide connectivity, low latency, high bandwidth • Many interconnection networks developed over last 2 decades • Hypercube • Mesh, torus • Ring, etc. CSE 160/Berman

  7. InterconnectionMedia … memories … Processors Basic Shared Memory Multiprocessor Shared Memory Communication • Processes in shared memory program communicate by accessing shared variables and data structures • Basic shared memory primitives • Read to a shared variable • Write to a shared variable CSE 160/Berman

  8. Shared variable x +1proc. A +1proc. B Accessing Shared Variables • Conflicts may arise if multiple processes want to write to a shared variable at the same time. • Programmer, language, and/or architecture must provide means of resolving conflicts Process A,B: read x compute x+1 write x CSE 160/Berman

  9. Architectural Support for SM • 4 basic types of interconnection media: • Bus • Crossbar switch • Multistage network • Interconnection network with distributed shared memory CSE 160/Berman

  10. … memory memory bus … processor processor Limited Scalability Media I • Bus • Bus acts as a “party line” between processors and shared memories • Bus provides uniform access to shared memory (UMA) • When bus saturates, performance of system degrades • For this reason, bus-based systems do not scale to more than 30-40 processors [Sequent Symmetry, Balance] CSE 160/Berman

  11. P1 P2 P3 P4 P5 M1 M2 M3 M4 M5 Limited Scalability Media II • Crossbar • Crossbar switch connects m processors and n memories with distinct paths between each processor/memory pair • Crossbar provides uniform access to shared memory (UMA) • O(mn) switches required for m processors and n memories • Crossbar scalable in terms of performance but not in terms of cost, used for basic switching mechanism in SP2 CSE 160/Berman

  12. P1 M1 P2 M2 Stage1 Stage2 Stagek P3 M3 … P4 M4 P5 M5 Multistage Networks • Multistage networks provide more scalable performance than bus but less costly to scale than crossbar • Typically max{logn,logm} stages connect n processors and m shared memories • “Omega” networks (butterfly, shuffle-exchange) commonly used for multistage network • Multistage network used for CM-5 (fat-tree connects processor/memory pairs), BBN Butterfly (butterfly), IBM RP3 (omega) CSE 160/Berman

  13. Butterfly multistage Used for BBN Butterfly, TC2000 Shuffle multistage Used for RP3, SP2 high performance switch 1 1 2 A 3 2 4 B A 3 B C C 4 D D Omega Networks CSE 160/Berman

  14. Fat-tree Interconnect • Bandwidth is increased towards the root • Used for data network for CM-5 (MIMD MPP) • 4 leaf nodes, internal nodes have 2 or 4 children • To route from leaf A to leaf B, pick random switch C in the least common ancestor fat node of A and B, take unique tree route from A to C and from C to B Binary fat-tree in which all internal nodes have two children CSE 160/Berman

  15. InterconnectionNetwork PM PM PM Processorand localmemory Sharedmemory … … Distributed Shared Memory • Memory is physically distributed but programmed as shared memory • Programmers find shared memory paradigm desirable • Shared memory distributed among processors, accesses may be sent as messages • Access to local memory and global shared memory creates NUMA (non-uniform memory access architectures) • BBN butterfly is NUMA shared memory multiprocessor BBNbutterflyinterconnect CSE 160/Berman

  16. Alphabet Soup • Terms for shared memory multiprocessors • NUMA = non-uniform memory access • BBN Butterfly, Cray T3E, Origin 2000 • UMA = uniform memory access • Sequent, Sun HPC 1000 • COMA = cache-only memory access • KSR • (NORMA = no remote memory access • message-passing MPPs ) CSE 160/Berman

  17. Using both SM and MP together • Common for platforms to support one model at a time – SM or MP • Clusters of SMPs may be effectively programmed using both SM and MP • SM used within a multiple processor machine/node • MP used between nodes CSE 160/Berman

  18. SM Program: Prefix Sums • Problem: Given n processes {P_i} and n datum {a_i}, want to compute the prefix sums {(a_1+…+ a_j )= A_1i} such that A_1i is in P_i upon termination of the algorithm. • We’ll look at an O(log n) SM parallel algorithm which computes the prefix sums of n datum on n processors CSE 160/Berman

  19. P1 P2 P3 P4 P5 P6 P7 P8 Initial values in shared memories a_1 a_2 a_3 a_4 a_5 a_6 a_7 a_8 A11 A11 A11 A12 A12 A12 A13 A13 A23 A14 A34 A14 A45 A15 A25 A36 A56 A16 A47 A17 A67 A58 A78 A18 Prefix sums in shared memories Data Movement for Prefix Sums Algorithm • Aij = a_i + a_i+1 + … + a_j CSE 160/Berman

  20. Aik A(k+1)j Aij Pseudo-code for Prefix Sums Algorithm • Pseudo-code Procedure ALLSUMS(a_1,…,a_n) Initialize P_i with data a_i=Aii for j=0 to (log n) –1 do forall i = 2^j +1 to n do (parallel for) Processor P_i: (i) obtains contents of P_i-2^j through shared memory and (ii) replaces contents of P_i with contents of P_i-2^j + current contents of P_i end forall end for CSE 160/Berman

  21. Programming Issues • Algorithm assumes that all additions with the same offset (i.e. for each level) are performed at the same time • Need some way of tagging or synchronizing computations • May be cost-effective to do a barrier synchronization (all processors must reach a “barrier before proceeding to the next level ) between levels • For this algorithm, there are no write conflicts within a level since one of the values is already in the shared variable, the other value need only be summed with the existing value • If two values must be written with existing variable, we would need to establish a well-defined protocol for handling conflicting writes CSE 160/Berman

  22. Send value from P1 Compare P1 and P2values, retain max ofP1 and P2 values in P2 Send min of values to P1 P1 P2 MP Program: Sorting • Problem: Sorting a list of numbers/keys (rearranging them so as to be in non-decreasing order) • Basic sorting operation: compare/exchange (compare/swap) • In serial computation (RAM) model, optimal sorting for n keys is O(nlogn) [1 “active”, 1 “passive” Processor] CSE 160/Berman

  23. Odd-Even Transposition Sort • Parallel version of bubblesort – many compare-exchanges done simultaneously • Algorithm consists of Odd Phases and Even Phases • In even phase, even-numbered processes exchange numbers (via messages) with their right neighbor • In odd phase, odd-numbered processes exchange numbers (via messages) with their right neighbor • Algorithm alternates odd phase and even phase for O(n) iterations CSE 160/Berman

  24. Odd-Even Transposition Sort • Data Movement P0 P1 P2 P3 P4 T=1 T=2 T=3 T=4 T=5 General Pattern for n=5 CSE 160/Berman

  25. Odd-Even Transposition Sort 3 10 4 8 1 T=0 • Example P0 P1 P2 P3 P4 T=1 3 10 4 8 1 T=2 3 4 10 1 8 T=3 3 4 1 10 8 T=4 3 1 4 8 10 T=5 1 3 4 8 10 General Pattern for n=5 CSE 160/Berman

  26. Odd-Even Transposition Code • Compare-exchange accomplished through message passing • Odd Phase • Even Phase P_i = 0, 2,4,…,n-2 recv(&A, P_i+1); send(&B, P_i+1); if (A<B) B=A; P_i = 1,3,5,…,n-1 send(&A, P_i-1); recv(&B, P_i-1); if (A<B) A=B; P0 P1 P2 P3 P4 P_i = 1,3,5,…,n-3 recv(&A, P_i+1); send(&B, P_i+1); if (A<B) B=A; P_i = 2,4,6,…,n-2 send(&A, P_i-1); recv(&B, P_i-1); if (A<B) A=B; P0 P1 P2 P3 P4 CSE 160/Berman

  27. Programming Issues • Algorithm that odd phases and even phases done in sequence – how to synchronize? • Synchronous execution • Need to have barrier between phases • Barrier synchronization costs may be high • Asynchronous execution • Need to tag iteration, phase so that correct values combined • Program may be implemented as SPMD (single program, multiple data) [see HW] CSE 160/Berman

  28. Programming Issues • Algorithm must be mapped to underlying platform • If communication costs >> computation costs, it may be more cost-effective to map multiple processes to a single processor and bundle communication • granularity (ratio of time required for a basic communication operation to the time required for a basic computation)of underlying platform required to determine best mapping Processor A Processor B Processor A Processor B P0 P1 P2 P3 P0 P1 P2 P3 P4 P4 CSE 160/Berman

  29. Is Odd-Even Transposition Sort Optimal? • What is optimal? • An algorithm is optimal if there is a lower bound for the problem it addresses with respect to the basic operation being counted which equals the upper bound given by the algorithm’s complexity function, i.e. lower bound = upper bound CSE 160/Berman

  30. Odd-Even Transposition Sort is optimal on linear array • Upper bound = O(n) • Lower bound = O(n) • Consider sorting algorithms on linear array where basic operation being counted is compare-exchange • If minimum key is in rightmost array element, it must move throughout the course of any algorithm to the leftmost array element • Compare-exchange operations only allow keys to move one process to the left each time-step. • Therefore, any sorting algorithm requires at least O(n) time-steps to move the minimum key to the first position 8 10 5 7 1 CSE 160/Berman

  31. Optimality • O(nlogn) lower bound for serial sorting algorithms on RAM wrt comparisons • O(n) lower bound for parallel sorting algorithms on linear array wrt compare-exchange • No conflict since the platforms/computing environments are different, apples vs. oranges • Note that in parallel world, different lower bounds for sorting in different environments • O(logn) lower bound on PRAM (Parallel RAM) • O(n^1/2) lower bound on 2D array, etc. CSE 160/Berman

  32. Y X Optimality on a 2D Array • Same argument as linear array works for lower bound • If we want to exchangeX and Y, we must do at least O( ) stepson an X array • upper bound:Thompson and Kung“Sorting on a Mesh-Connected Parallel Computer” CACM, Vol 20, (April), 1977 CSE 160/Berman

More Related