Spark

1. Spark A framework for iterative and interactive cluster computing Matei Zaharia, MosharafChowdhury, Michael Franklin, Scott Shenker, Ion Stoica

2. Outline Background: Nexus project Spark goals Programming model Example jobs Implementation Interactive Spark

3. Nexus Background Rapid innovation in cluster computing frameworks Pregel Apache Hama Dryad Pig

4. Problem Rapid innovation in cluster computing frameworks No single framework optimal for all applications Want to run multiple frameworks in a single cluster • …to maximize utilization • …to share data between frameworks • …to isolate workloads

5. Solution Nexus is an “operating system” for the cluster over which diverse frameworks can run • Nexus multiplexes resources between frameworks • Frameworks control job execution

6. Nexus Architecture Hadoop job Hadoop job MPI job Hadoop v19 scheduler Hadoop v20 scheduler MPI scheduler Nexus master Nexus slave Nexus slave Nexus slave MPI executor MPI executor Hadoop v19 executor Hadoop v20 executor Hadoop v19 executor task task task task task

7. Nexus Status Prototype in 7000 lines of C++ Ported frameworks: • Hadoop(900 line patch) • MPI (160 line wrapper scripts) New frameworks: • Spark,Scala framework for iterative jobs & more • Apache+haproxy, elastic web server farm (200 lines)

8. Outline Background: Nexus project Spark goals Programming model Example job Implementation Interactive Spark

9. Spark Goals Support iterative jobs • Machine learning researchers in our lab identified this as a workload that Hadoop doesn’t perform well on Experiment with programmability • Leverage Scala to integrate cleanly into programs • Support interactive use from Scala interpreter Retain MapReduce’s fine-grained fault-tolerance

10. Programming Model Distributed datasets • HDFS files, “parallelized” Scalacollections • Can be transformed with map and filter • Can be cached across parallel operations Parallel operations • Foreach, reduce, collect Shared variables • Accumulators (add-only) • Broadcast variables (read-only)

11. Example 1:Logistic Regression

12. Logistic Regression Goal: find best line separatingtwo sets of points random initial line + + + + + + – + + – – + – + – – – – – – target

13. Serial Version val data = readData(...) varw = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = Vector.zeros(D) for (p <- data) { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x } w -= gradient } println("Finalw: " + w)

14. Spark Version val data = spark.hdfsTextFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(Vector.zeros(D)) for (p <- data) { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x } w -= gradient.value } println("Final w: " + w)

15. Spark Version val data = spark.hdfsTextFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(Vector.zeros(D)) for (p <- data) { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x } w -= gradient.value } println("Final w: " + w)

16. Spark Version val data = spark.hdfsTextFile(...).map(readPoint).cache() varw = Vector.random(D) for (i <- 1 to ITERATIONS) { var gradient = spark.accumulator(Vector.zeros(D)) data.foreach(p => { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y gradient += scale * p.x }) w -= gradient.value } println("Finalw: " + w)

17. Functional Programming Version val data = spark.hdfsTextFile(...).map(readPoint).cache() varw = Vector.random(D) for (i <- 1 to ITERATIONS) { w -= data.map(p => { val scale = (1/(1+exp(-p.y*(w dot p.x))) - 1) * p.y scale * p.x }).reduce(_+_) } println("Finalw: " + w)

18. Job Execution updateparam aggregate Master param Big Dataset R2 R3 R4 R1 Slave 1 Slave 2 Slave 3 Slave 4 Spark

19. Job Execution updateparam Master aggregate Master param param param aggregate Reduce R2 R3 R4 R1 aggregate Map 5 Map 6 Map 7 Map 8 Slave 1 Slave 2 Slave 3 Slave 4 Map 1 Map 2 Map 3 Map 4 Reduce  Spark Hadoop / Dryad

20. Performance 127 s / iteration first iteration 174 s further iterations 6 s

21. Example 2:Alternating Least Squares

22. Collaborative Filtering Predict movie ratings for a set of users based on their past ratings 1 ? ? 4 5 ? 3 ? ? 3 5 ? ? 3 5 ? 5 ? ? ? 1 4 ? ? ? ? 2 ? R = Users Movies

23. Matrix Factorization Model R as product of user and movie matrices A and B of dimensions U×K and M×K R A BT = Problem: given subset of R, optimize A and B

24. Alternating Least Squares Algorithm Start with random A and B Repeat: • Fixing B, optimize A to minimize error on scores in R • Fixing A, optimize B to minimize error on scores in R

25. Serial ALS val R = readRatingsMatrix(...) var A = (0 until U).map(i => Vector.random(K)) var B = (0 untilM).map(i => Vector.random(K)) for (i <- 1 to ITERATIONS) { A = (0 until U).map(i => updateUser(i, B, R)) B = (0 untilM).map(i => updateMovie(i, A, R)) }

26. Naïve Spark ALS val R = readRatingsMatrix(...) var A = (0 until U).map(i => Vector.random(K)) var B = (0 untilM).map(i => Vector.random(K)) for (i <- 1 to ITERATIONS) { A = spark.parallelize(0 until U, numSlices) .map(i => updateUser(i, B, R)) .collect() B =spark.parallelize(0 until M, numSlices) .map(i => updateMovie(i, A, R)) .collect() } Problem:R re-sent to all nodes in each parallel operation

27. Efficient Spark ALS val R = spark.broadcast(readRatingsMatrix(...)) var A = (0 until U).map(i => Vector.random(K)) var B = (0 untilM).map(i => Vector.random(K)) for (i <- 1 to ITERATIONS) { A = spark.parallelize(0 until U, numSlices) .map(i => updateUser(i, B, R.value)) .collect() B =spark.parallelize(0 until M, numSlices) .map(i => updateMovie(i, A, R.value)) .collect() } Solution: mark R as broadcast variable

28. ALS Performance

29. Subseq. Iteration Breakdown 36% of iteration spent on broadcast

30. Outline Background: Nexus project Spark goals Programming model Example job Implementation Interactive Spark

31. Architecture local cache Driver program connects to Nexus and schedules tasks Workers run tasks, report results and variable updates Data shared with HDFS/NFS No communication between workers for now user code, broadcast vars HDFS Nexus Driver tasks, results Workers

32. Distributed Datasets Each distributed dataset object maintains a lineage that is used to rebuild slices that are lost / fall out of cache Ex: errors = textFile(“log”).filter(_.contains(“error”)) .map(_.split(‘\t’)(1)) .cache() getIterator(slice) HdfsFile path: hdfs://… FilteredFile func: contains(...) MappedFile func: split(…) CachedFile HDFS Local cache

33. Language Integration Scala closures are Serializable objects • Serialize on driver, load & run on workers Not quite enough • Nested closures may reference entire outer scope • May pull in non-Serializablevariables not used inside • Solution: bytecode analysis + reflection Shared variables • Accumulators: serialized form contains ID • Broadcast vars: serialized form is path to HDFS file

34. Interactive Spark Modified Scala interpreter to allow Spark to be used interactively from the command line Required two changes: • Modified wrapper code generation so that each “line” typed has references to objects for its dependencies • Place generated classes in distributed filesystem Enables in-memory exploration of big data

35. Demo

36. Conclusions Spark provides two abstractions that enable iterative jobs and interactive use: • Distributed datasets with controllable persistence, supporting fault-tolerant parallel operations • Shared variables for efficient broadcast and imperative style programming Language integration achieved using Scala features + some amount of hacking All this is surprisingly little code (~1600 lines)

37. Related Work DryadLINQ • SQL-like queries integrated in C# programs • Build queries through operations on lazy datasets • Cannot have a dataset persist across queries • No concept of shared variables for broadcast etc Pig & Hive • Query languages that can call into Java/Python/etc UDFs • No support for caching a dataset across queries OpenMP • Compiler extension for parallel loops in C++ • Annotate variables as read-only or accumulator above loop • Cluster version exists, but not fault-tolerant

38. Future Work Open-source Spark and Nexus • Probably this summer • Very interested in getting users! Understand which classes of algorithms we can handle and how to extend Spark for others Build higher-level interfaces on top of interactive Spark (e.g. R, SQL)

39. Questions ? ? ?