Semi-Explicit Parallel Programming in Haskell

1. Semi-Explicit Parallel Programming in Haskell Satnam Singh Microsoft Research Cambridge Leeds2009

2. 0 1 19

3. 9 0 1 19

4. publicclassArraySummer { privatedouble[] a; // Encapsulated array privatedouble sum; // Variable used to compute sum // Constructor requiring an initial value for array publicArraySummer(double[] values) { a = values; } // Method to compute the sum of segment of the array publicvoidSumArray(intfromIndex, inttoIndex, outdoublearraySum) { sum = 0; for (inti = fromIndex; i < toIndex; i++) sum = sum + a[i]; arraySum = sum; } }

5. ThreadCreate thread.Start thread 1 thread 2 thread.Join

6. classProgram { staticvoid Main(string[] args) { constinttestSize = 100000000; double[] testValues = newdouble[testSize] ; for (inti = 0; i < testSize; i++) testValues[i] = i/testSize; ArraySummer summer = newArraySummer(testValues) ; StopwatchstopWatch = newStopwatch(); stopWatch.Start(); doubletestSum ; summer.SumArray(0, testSize, outtestSum); TimeSpants = stopWatch.Elapsed; Console.WriteLine("Sum duration (mili-seconds) = " + stopWatch.ElapsedMilliseconds); Console.WriteLine("Sum value = " + testSum); Console.ReadKey(); } } }

7. classProgram { staticvoid Main(string[] args) { constinttestSize = 100000000; double[] testValues = newdouble[testSize]; for (inti = 0; i < testSize; i++) testValues[i] = i / testSize; ArraySummer summer = newArraySummer(testValues); StopwatchstopWatch = newStopwatch(); stopWatch.Start(); doubletestSumA = 0 ; doubletestSumB; ThreadsumThread = newThread(delegate() { summer.SumArray(0, testSize / 2, outtestSumA); }); sumThread.Start(); summer.SumArray(testSize/2+1, testSize, outtestSumB); sumThread.Join(); TimeSpants = stopWatch.Elapsed; Console.WriteLine("Sum duration (mili-seconds) = " + stopWatch.ElapsedMilliseconds); Console.WriteLine("Sum value = " + (testSumA+testSumB)); Console.ReadKey(); } }

8. The Accidental Semi-colon ;

9. A ; B ; createThread (A) ; B; A A B B

10. Execution Model “Thunk” for “fib 10” Pointer to the implementation • 1 • 1 Values for free variables • 8 • 10 Storage slot for the result • 9 • fib 0 = 0 • fib 1 = 1 • fib n = fib (n-1) + fib (n-2) • 5 • 8 • 5 • 8 • 3 • 6

11. wombat and numbat wombat :: Int -> Int wombat n = 42*n numbat :: Int -> IOInt numbat n = do c <- getChar return (n + ord c) pure function side-effecting function Computation inside a ‘monad’

12. IO (), pronounced “IO unit” numbat :: IO () numbat = do c <- getChar putChar (chr (1 + ord c))

13. f (g + h) z!!2 mapM f [a, b, ... , g] infer type [Int] -> Bool IO String pure function deterministic stateful operation may be non-deterministic

14. Functional Programming to the Rescue? • Why not evaluate every-sub expression of our pure functional programs in parallel? • execute each sub-expression in its own thread? • The 80s dream does not work: • granularity • data-dependency

15. Infix Operators • mod a b mod 7 3 = 1 • Infix with backquotes: a `mod` b 7 `mod` 3 = 1

16. x `par` y • x is sparked for speculative evaluation • a spark can potentially be instantiated on a thread running in parallel with the parent thread • x `par` y = y • typically x used inside y • blurRows `par` (mix blurColsblurRows)

17. x `par` (y + x) y y is evaluated first x is evaluated second x x x is sparked x fizzles

18. x `par` (y + x) P1 P2 y y is evaluated on P1 x x is taken up for evaluation on P2 x x is sparked on P1

23. par is Not Enough • pseq :: a -> b -> b • pseqis strict in its first argument but not in its second argument • Related function: • seq :: a -> b -> b • Strict in both arguments • Compiler may transform seq x y to seqy x • No good for controlling order for evaluation for parallel programs

26. Don Stewart Parallel fib with threshold cutoff = 35 -- Threshold for parallel evaluation -- Sequential fib fib' :: Int -> Integer fib' 0 = 0 fib' 1 = 1 fib' n = fib' (n-1) + fib' (n-2) -- Parallel fib with thresholding fib :: Int -> Integer fib n | n < cutoff = fib' n | otherwise = r `par` (l `pseq` l + r) where l = fib (n-1) r = fib (n-2) -- Main program main = forM_ [0..45] $ \i -> printf "n=%d => %d\n" i (fib i)

27. Parallel fib performance

28. Parallel quicksort (wrong) quicksortN :: (Ord a) => [a] -> [a] quicksortN [] = [] quicksortN [x] = [x] quicksortN (x:xs) = losort `par` hisort `par` losort ++ (x:hisort) where losort = quicksortN [y|y <- xs, y < x] hisort = quicksortN [y|y <- xs, y >= x]

29. What went wrong? cons cell losort Unevaluated thunk Unevaluated thunk

30. forceList forceList :: [a] -> () forceList [] = () forceList (x:xs) = x `seq` forceListxs

31. Parallel quicksort (right) quicksortF [] = [] quicksortF [x] = [x] quicksortF (x:xs) = (forceListlosort) `par` (forceListhisort) `par` losort ++ (x:hisort) where losort = quicksortF [y|y <- xs, y < x] hisort = quicksortF [y|y <- xs, y >= x]

32. parSumArray :: Array Int Double -> Double parSumArray matrix = lhs `par` (rhs`pseq` lhs + rhs) where lhs = seqSum 0 (nrValues `div` 2) matrix rhs = seqSum (nrValues `div` 2 + 1) (nrValues-1) matrix

33. Strategies • Haskell provides a collection of evaluation strategies for controlling the evaluation order of various data-types. • Users have to define indicate how their own types are evaluated to a normal form. • Algorithms + Strategy = Parallelism, P. W. Trinder, K. Hammond, H.-W. Loidl and S. L. Peyton Jones. • http://www.macs.hw.ac.uk/~dsg/gph/papers/html/Strategies/strategies.html

34. Explicitly Creating Threads • forkIO :: IO () -> ThreadID • Creates a lightweight Haskell thread, not an operating system thread.

35. Inter-thread Communication • putMVar :: MVar a -> IO () • takeMVar :: MVar a -> IO a

36. MVars empty 52 mv ... putMVarmv 52 ... ... ... ... v <- takeMVarmv ...

37. Rendezvous threadA :: MVarInt -> MVar Float -> IO () threadAvalueToSendMVarvalueReceivedMVar = do -- some work -- new perform rendezvous by sending 72 putMVarvalueToSendMVar 72 -- send value v <- takeMVarvalueToReadMVar putStrLn (show v)

38. Rendezvous threadB :: MVarInt -> MVar Float -> IO () threadBvalueToReceiveMVarvalueToSendMVar = do -- some work -- now perform rendezvous by waiting on value z <- takeMVarvalueToReceiveMVar putMVarvalueToSendMVar (1.2 * z) -- continue with other work

39. Rendezvous main :: IO () main = do aMVar <- newEmptyMVar bMVar <- newEmptyMVar forkIO (threadAaMVarbMVar) forkIO (threadBaMVarbMVar) threadDelay 1000 -- BAD!

40. fib again fib :: Int -> Int -- As before fibThread :: Int -> MVarInt -> IO () fibThread n resultMVar = putMVarresultMVar (fib n) sumEuler :: Int -> Int -- As before

41. fib fixed fibThread :: Int -> MVarInt -> IO () fibThread n resultMVar = do pseq f (return ()) putMVarresultMVar f where f = fib n

42. $ time fibForkIO +RTS -N1 real 0m40.473s user 0m0.000s sys 0m0.031s $ time fibForkIO +RTS -N2 real 0m38.580s user 0m0.000s sys 0m0.015s

43. “STM”s in Haskell data STM a instance Monad STM -- Monads support "do" notation and sequencing -- Exceptions throw :: Exception -> STM a catch :: STM a -> (Exception->STM a) -> STM a -- Running STM computations atomically :: STM a -> IO a retry :: STM a orElse :: STM a -> STM a -> STM a -- Transactional variables data TVar a newTVar :: a -> STM (TVar a) readTVar :: TVar a -> STM a writeTVar :: TVar a -> a -> STM ()