1 / 34

Parallel Patterns Reduce & Scan

Parallel Patterns Reduce & Scan. Programming Patterns For Parallelism. Some patterns repeat in many different contexts e.g. Search an element in an array Identifying such patterns important Solve a problem once and reuse the solution Split a hard problem into individual problems

hila
Download Presentation

Parallel Patterns Reduce & Scan

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Parallel Patterns - Reduce & Scan Parallel PatternsReduce & Scan

  2. Parallel Patterns - Reduce & Scan Programming Patterns For Parallelism • Some patterns repeat in many different contexts • e.g. Search an element in an array • Identifying such patterns important • Solve a problem once and reuse the solution • Split a hard problem into individual problems • Helps define interfaces

  3. Parallel Patterns - Reduce & Scan We Have Already Seen Some Patterns

  4. Parallel Patterns - Reduce & Scan We Have Already Seen Some Patterns • Divide and Conquer • Split a problem into n sub problems • Recursively solve the sub problems • And merge the solution • Data Parallelism • Apply the same function to all elements in a collection, array • Parallel.For, Parallel.ForEach • Also called as “map” in functional programming

  5. Parallel Patterns - Reduce & Scan Map • Given a function f : (A) => B • A collection a: A[] • Generates a collection b: B[], where B[i] = f( A[i] ) • Parallel.For, Paralle.ForEach • Where each loop iteration is independent A f f f f f f f f B

  6. Parallel Patterns - Reduce & Scan Reduce And Scan • In practice, parallel loops have to work together to generate an answer • Reduce and Scan patterns capture common cases of processing results of Map

  7. Parallel Patterns - Reduce & Scan Reduce And Scan • In practice, parallel loops have to work together to generate an answer • Reduce and Scan patterns capture common cases of processing results of Map • Note: Map and Reduce are similar to but not the same as MapReduce • MapReduce is a framework for distributed computing

  8. Parallel Patterns - Reduce & Scan Reduce • Given a function f: (A, B) => B • A collection a: A[] • An initial value b0: B • Generate a final value b: B • Where b = f(A[n-1], … f(A[1], f(A[0], b0)) ) A b b0 f f f f f f f f

  9. Parallel Patterns - Reduce & Scan Reduce • Given a function f: (A, B) => B • A collection a: A[] • An initial value b0: B • Generate a final value b: B • Where b = f(A[n-1], … f(A[1], f(A[0], b0)) ) • Only consider where A and B are the same type A b b0 f f f f f f f f

  10. Parallel Patterns - Reduce & Scan Reduce B acc = b_0; for( i = 0; i < n; i++ ) { acc = f( a[i], acc ); } b = acc; A b b0 f f f f f f f f

  11. Parallel Patterns - Reduce & Scan Associativity of the Reduce function • Reduce is parallelizable if f is associative f(a, f(b, c)) = f(f(a,b), c) • E.g. Addition : (a + b) + c = a + (b + c) • Where + is integer addition (with modulo arithmetic) • But not when + is floating point addition

  12. Parallel Patterns - Reduce & Scan Associativity of the Reduce function • Reduce is parallelizable if f is associative f(a, f(b, c)) = f(f(a,b), c) • E.g. Addition : (a + b) + c = a + (b + c) • Where + is integer addition (with modulo arithmetic) • But not when + is floating point addition • Max, min, multiply, … • Set union, intersection,

  13. Parallel Patterns - Reduce & Scan We can use Divide and Conquer • Reduce(f, A[1…n], b_0) = f ( Reduce(f, A[1..n/2], b_0), Reduce(f, A[n/2+1…n], I) ) where I is the identity element of f A f f f f f f f f b0 I b f

  14. Parallel Patterns - Reduce & Scan Implementation Optimizations • Switch to sequential Reduce for the base k elements • Do k way splits instead of two way splits • Maintain a thread-local accumulated value • A task updates the value of the thread it executes in

  15. Parallel Patterns - Reduce & Scan Implementation Optimizations • Switch to sequential Reduce for the base k elements • Do k way splits instead of two way splits • Maintain a thread-local accumulated value • A task updates the value of the thread it executes in • Requires that the reduce function is also commutative f(a, b) = f(b, a)

  16. Parallel Patterns - Reduce & Scan Implementation Optimizations • Switch to sequential Reduce for the base k elements • Do k way splits instead of two way splits • Maintain a thread-local accumulated value • A task updates the value of the thread it executes in • Requires that the reduce function is also commutative f(a, b) = f(b, a) • Thread local values are then merged in a separate pass

  17. Parallel Patterns - Reduce & Scan Scan • Given a function f: (A, B) => B • A collection a: A[] • An initial value b0: B • Generate a collection b: B[] • Where b[i] = f(A[i-1], … f(A[1], f(A[0], b0)) ) A b0 f f f f f f f f

  18. Parallel Patterns - Reduce & Scan Scan B acc = b_0; for( i = 0; i < n; i++ ) { acc = f( a[i], acc ); } A b0 f f f f f f f f

  19. Parallel Patterns - Reduce & Scan Scan is Efficiently Parallelizable • When f is associative

  20. Parallel Patterns - Reduce & Scan Scan is Efficiently Parallelizable • When f is associative • Scan(f, A[1..n], b_0) = Scan(f, A[1..n/2], b_0), Scan(f, A[n/2+1…n], ____) A ? b0 f f f f f f f f

  21. Parallel Patterns - Reduce & Scan Scan is Efficiently Parallelizable • When f is associative • Scan(f, A[1..n], b_0) = Scan(f, A[1..n/2], b_0), Scan(f, A[n/2+1…n], Reduce(f, A[1..n/2], b_0)) A ? b0 f f f f f f f f

  22. Parallel Patterns - Reduce & Scan Scan is useful in many places • Radix Sort • Ray Tracing • …

  23. Parallel Patterns - Reduce & Scan Scan is useful in many places • Radix Sort (  ) • Ray Tracing • …

  24. Parallel Patterns - Reduce & Scan Computing Line of Sight • Given x1, … xn with altitudes a[1],…a[n] • Which of the points are visible from x0

  25. Parallel Patterns - Reduce & Scan Computing Line of Sight • Given x0, … xn with altitudes alt[0],…alt[n] • Which of the points are visible from x0 • angle[i] = arctan( (alt[i] – alt[0]) / i ) • xi is visible from x0 if all points between them have lesser angle than angle[i]

  26. Parallel Patterns - Reduce & Scan Solution

  27. Parallel Patterns - Reduce & Scan Radix Sort

  28. Parallel Patterns - Reduce & Scan Radix Sort

  29. Parallel Patterns - Reduce & Scan Radix Sort

  30. Parallel Patterns - Reduce & Scan Radix Sort

  31. Parallel Patterns - Reduce & Scan Basic Primitive: Pack • Given an array A and an array F of flags • A = [5 7 2 4 5 3 1] • F = [1 1 0 0 1 1 1] • Pack all elements with flag = 0 before elements with flag = 1 • A’ = [2 4 5 7 5 3 1]

  32. Parallel Patterns - Reduce & Scan Solution

  33. Parallel Patterns - Reduce & Scan Other Applications of Scan • Radix Sort • Computing Line of Sight • Adding multi-precision numbers • Quick Sort • To search for regular expressions • Parallel grep • …

  34. Parallel Patterns - Reduce & Scan High Level Points • Minimize dependence between parallel loops • Unintended dependences = data races • Next lecture • Carefully analyze remaining dependences • Use Reduce and Scan patterns where applicable

More Related