CS51: Introduction to Parallelism

1. CS51: Introduction to Parallelism Greg Morrisett

2. Topic for Today: Parallelism • What is it? • Why is it particularly important today? • How can we take advantage of it? • Why is it so much harder to program? • Some linguistic constructs • thread creation • thread coordination: locks & conditions • thread coordination: communication channels

3. Parallelism • What is it? • doing many things at the same time instead of sequentially (one-after-the-other).

4. Flavors of Parallelism • Data Parallelism • same computation being performed on a lot of data • e.g., adding two vectors of numbers • Task Parallelism • different computations/programs running at the same time • e.g., running web server and database • Pipeline Parallelism • assembly line • e.g., parse function, type-check function, translate to intermediate representation, optimize representation, register allocate, code generate.

5. Parallelism vs. Concurrency • No consistent use of these terms. • For our discussion: • Parallelism: speeding up an application: • decrease total running time of program by utilizing additional resources. • e.g., running your web-crawler on a cluster versus one machine. • Concurrency: effectively utilizing resources: • example: running your clock, login service, editor, compiler, chat, etc. all at the same time on a single CPU. • OS gives each of these programs a small time-slice (~10msec) • often slows throughput due to cost of switching contexts

6. Parallelism Why is it particularly important (today)? • In one word: POWER

7. CPU Clock Speeds from 1993-2005

8. CPU Clock Speeds from 1993-2005 No need to get a multiprocessor when next year’s machine is twice as fast!

9. CPU Clock Speeds from 1993-2005 Oops!

10. CPU Power 1993-2005

11. CPU Power 1993-2005 But power consumption is only part of the problem…cooling is the other!

12. The Heat Problem

13. The Problem 2005 Cooler 1993 Pentium Heat Sink

14. Today: water cooled!

15. Actually, Cray-4: 1994 Up to 64 processors Running at 1 GHz 8 Megabytes of RAM Cost: roughly $10M The CRAY 2,3, and 4 CPU and memory boards were immersed in a bath of electricallyinert cooling fluid.

16. Power Dissipation

17. Parallelism Why is it particularly important (today)? • Roughly every other year, a chip from Intel would: • halve the feature size (size of transistors, wires, etc.) • double the number of transistors • double the clock speed • this drove the economic engine of the IT industry (and the US!) • No longer able to double clock or cut voltage: a processor won’t get any faster! • (so why should you buy a new laptop, desktop, etc.?) • power and heat are limitations on the clock • errors, variability (noise) are limitations on the voltage • but we can still pack a lot of transistors on a chip… (at least for another 10 to 15 years.)

18. So… • Instead of trying to make your CPU go faster, Intel’s just going to pack more CPUs onto a chip. • last year: dual core (2 CPUs). • this year: quad core (4 CPUs). • Intel is testing 48-core chips now. • Within 10 years, you’ll have ~1024 Intel CPUs on a chip. • In fact, that’s already happening with graphics chips (Nvidia, ATI). • really good at simple data parallelism (many deep pipes) • but they are much dumber than an Intel core. • and right now, chew up a lot of power. • watch for GPUs to get “smarter” and more power efficient, while CPUs become more like GPUs.

19. Example

20. Nvidia GTX 590 Card

21. Sounds Great! So my programs will run 2x, 4x, 48x, 256x, 1024x faster?

22. Sounds Great! • So my programs will run 2x, 4x, 48x, 256x, 1024x faster? • afraid not.

23. Sounds Great! • So my programs will run 2x, 4x, 48x, 256x, 1024x faster? • afraid not. • not every program can be effectively parallelized. • in fact, very few programs will scale with perfect speedups. • rather, they are limited by the portions which must be sequential. • see Amdahl’s law. • parallel code requires coordination or synchronization. • for example, in a pipeline, we must have gates to control the flow of data through the pipe. Opening/closing those gates adds overhead. • on your chip, the synchronization is controlled by a global clock. • known as synchronous coordination • but most software is controlled by asynchronous coordination • i.e., without the help of a global clock

24. Most Troubling… Parallel code is hard to get right. • New classes of errors: • race conditions • deadlocks • New classes of performance/scalability problems: • busy-waiting • sequentialization • contention Programmers tend to get both things wrong  And testing becomes a nightmare…

25. Solid Parallel Programming Requires much more than we’ll have time for in this class. • Good baseline programming skills. • all the things we’ve been talking about in here • Deep knowledge of the application. • often, requires changing the problem or using some structure that’s specific to the application (e.g., n-body problems). • Discipline in following safe programming patterns. • e.g., make all data accesses protected by appropriate locks • e.g., carefully acquire locks in the right order • e.g., use patterns that are robust to parallelism (e.g., immutable data.) • Careful engineering to ensure scaling. • e.g., fine-grained locking (at odds with 3 above) • matching the parallelism in the problem to the resources • careful data movement/communication

26. From the Programming Standpoint • Threads: an abstraction of a processor. • programmer (or compiler) decides that some work can be done in parallel with some other work, e.g.: • let x = compute_big_thing() inlet y = compute_other_big_thing() in ... • we fork a thread to run the computation in parallel, e.g.: • lett = Thread.createcompute_big_thing () inlet y = compute_other_big_thing() in...

27. In Pictures (abstractly) let t = Thread.createf () inlet y = g () in ... Processor 1Processor 2 t0 Thread.create doing nothing t1 runs g() runs f() t2 … …

28. Of Course… • What happens when you have 2 cores and you fork 4 threads to run at the same time? • the operating system provides the illusion that there are an infinite number of processors. • (not really: each thread consumes space, so if we fork too many threads the process will die.) • it then time-multiplexes the threads across the available processors. • about every 10 msec, it stops the current thread on a processor, and switches to another thread. • so a thread is really a virtual processor. • In reality, Ocaml threads do not support multiple cores at all! • instead, it only time-multiplexes on one core • so we won’t be seeing any speedups from multi-threading • but we will see all of the problems of parallel programming

29. Coordination Thread.create : (‘a -> ‘b) -> ‘a -> Thread.t lett = Thread.createf () inlety = g() in ... How do we get back the result that t is computing?

30. First Attempt let r = ref Nonelet t = Thread.create(fun _ -> r := Some(f ())) inlety = g() in match r with | Some v -> (* compute with v and y *) | None -> ???

31. Second Attempt letr = ref Nonein let t = Thread.create(fun _ -> r := Some(f ())) in lety = g() in let rec wait() = match r with | Some v -> v | None -> wait()let v = wait() in(* compute with v and y *)

32. Two Problems let r = ref Nonelet t = Thread.create(fun _ -> r := Some(f ())) inlety = g() in let rec wait() = match r with | Some v -> v | None -> wait()let v = wait() in(* compute with v and y *) First, we are busy-waiting. • consuming time without doing something useful. • the processor could be either running a useful thread/program or power down.

33. Two Problems let r = ref Nonelet t = Thread.create(fun _ -> r := Some(f ())) inlety = g() in let rec wait() = match r with | Some v -> v | None -> wait()let v = wait() in(* compute with v and y *) Second, an operation like “r := Some v” may not be atomic. • r := Some v requires us to copy the bytes of Some v into the ref r • we might see part of the bytes (corresponding to Some) before we’ve written in the other parts (e.g., v). • So the waiter might see the wrong value.

34. Atomicity Consider the following: let inc(r:intref) = r := (!r) + 1 and suppose two threads are incrementing the same ref r: Thread 1Thread 2 inc(r); inc(r); !r !r If r initially holds 0, then what will Thread 1 see when it reads r?

35. Atomicity The problem is that we can’t see exactly what instructions the compiler might produce to execute the code. It might look like this: Thread 1Thread 2 EAX := load(r); EAX := load(r); EAX := EAX + 1; EAX := EAX + 1; store EAX into r store EAX into r EAX := load(r) EAX := load(r)

36. Atomicity But a clever compiler might optimize this to: Thread 1Thread 2 EAX := load(r); EAX := load(r); EAX := EAX + 1; EAX := EAX + 1; store EAX into r store EAX into r EAX := load(r)EAX := load(r)

37. Atomicity Furthermore, we don’t know when the OS might interrupt one thread and run the other. Thread 1Thread 2 EAX := load(r); EAX := load(r); EAX := EAX + 1; EAX := EAX + 1; store EAX into r store EAX into r EAX := load(r) EAX := load(r) (The situation is similar, but not quite the same on multi-processor systems.)

38. Interleaving & Race Conditions • We can calculate the possible outcomes for a multi-threaded program by considering all of the possible interleavings of the atomic actions performed by each thread. • Subject to the happens-before relation. • can’t have a child thread’s actions happening before a parent forks it. • can’t have later instructions execute earlier in the same thread. • Here, atomic means indivisible actions. • For example, on most machines reading or writing a 32-bit word is atomic. • But, writing a multi-word object is usually not atomic. • Most operations like “b := b - w” are implemented in terms of a series of simpler operations such as “r1 = read(b); r2 = read(w); r3 = r1 – r2; write(b, r3)” • To better understand what is and isn’t atomic demands detailed knowledge of the compiler and the underlying architecture (see CS61, CS161 for this kind of detail.) • Reasoning about all interleavings is hard. • The number of interleavings grows exponentially with the number of statements. • It’s hard for us to tell what is and isn’t atomic in a high-level language.

39. Atomicity One possible interleaving of the instructions: Thread 1Thread 2 EAX := load(r); EAX := load(r); EAX := EAX + 1; EAX := EAX + 1; store EAX into r store EAX into r EAX := load(r) EAX := load(r) What answer do we get?

40. Atomicity Another possible interleaving: Thread 1Thread 2 EAX := load(r); EAX := load(r); EAX := EAX + 1; EAX := EAX + 1; store EAX into r store EAX into r EAX := load(r) EAX := load(r) What answer do we get this time?

41. Atomicity In fact, today’s multi-core processors don’t treat memory in a sequentially consistent fashion. Thread 1Thread 2 EAX := load(r); EAX := load(r); EAX := EAX + 1; EAX := EAX + 1; store EAX into r store EAX into r EAX := load(r) EAX := load(r) That means that we can’t even assume that what we will see corresponds to some interleaving of the threads’ instructions. (Beyond the scope of this class…)

42. One Solution (using join) let r = ref None let t = Thread.create(fun _ -> r := Some(f ())) inlety = g() in Thread.join t ; match r with | Some v -> (* compute with v and y *) | None -> failwith “impossible”

43. One Solution (using join) Thread.join t causes the current thread to wait until the thread t terminates. let r = ref None let t = Thread.create(fun _ -> r := Some(f ())) inlety = g() in Thread.join t ; match r with | Some v -> (* compute with v and y *) | None -> failwith “impossible”

44. One Solution (using join) So after the join, we know that any of the operations of t have completed. let r = ref None let t = Thread.create(fun _ -> r := Some(f ())) inlety = g() in Thread.join t ; match r with | Some v -> (* compute with v and y *) | None -> failwith “impossible”

45. Futures This pattern is so common, we’ll create an abstraction for it: module type FUTURE= sig type ‘a future (* future f x forks a thread to run f(x) and returns a future which can be used to synchronize with the thread and get the result. *)valfuture : (’a->’b) -> ’a -> ‘b future (* force f causes us to wait until the thread computing the future value is done and then returns its value. *) valforce :’a future -> ‘a end

46. Future Implementation module Future : FUTURE = struct type ‘a future = {tid:Thread.t ; value:’a option ref} let future(f:’a->’b)(x:’a) : ‘b future = let r = ref None inlet t = Thread.create (fun () -> r := Some(f x)) () in {tid=t ; value=r} let force (f:’a future) : ‘a = Thread.joinf.tid ; match !(f.value) with | Some v -> v | None -> failwith “impossible!” end

47. Now using Futures: let x = future f () inlet y = g () inlet v = force x in(* compute with v and y *)

48. In Pictures Thread 1t = fork f xinst1,1; inst1,2; inst1,3; inst1,4; … inst1,n-1; inst1,n; join t Thread 2 inst2,1; inst2,2; inst2,3; … inst2,m; We know that for each thread the previous instructions must happen before the later instructions. So for instance, inst1,1 must happen before inst1,2.

49. In Pictures Thread 1t = fork f xinst1,1; inst1,2; inst1,3; inst1,4; … inst1,n-1; inst1,n; join t Thread 2 inst2,1; inst2,2; inst2,3; … inst2,m; We also know that the fork must happen before the first instruction of the second thread.

50. In Pictures Thread 1t = fork f xinst1,1; inst1,2; inst1,3; inst1,4; … inst1,n-1; inst1,n; join t Thread 2 inst2,1; inst2,2; inst2,3; … inst2,m; We also know that the fork must happen before the first instruction of the second thread. And thanks to the join, we know that all of the instructions of the second thread must be completed before the join finishes.