1 / 140

Scala Parallel Collections

Scala Parallel Collections. Aleksandar Prokopec EPFL. Scala collections. for { s <- surnames n <- names if s endsWith n } yield (n, s). Mc Donald. Scala collections. for { s <- surnames n <- names if s endsWith n } yield (n, s). 1040 ms.

reece
Download Presentation

Scala Parallel Collections

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Scala Parallel Collections AleksandarProkopec EPFL

  2. Scala collections for { s <- surnames n <- names if s endsWith n } yield (n, s) McDonald

  3. Scala collections for { s <- surnames n <- names if s endsWith n } yield (n, s) 1040 ms

  4. Scala parallel collections for { s <- surnames n <- names if s endsWith n } yield (n, s)

  5. Scala parallel collections for { s <- surnames.par n <- names.par if s endsWith n } yield (n, s)

  6. Scala parallel collections for { s <- surnames.par n <- names.par if s endsWith n } yield (n, s) 2 cores 575 ms

  7. Scala parallel collections for { s <- surnames.par n <- names.par if s endsWith n } yield (n, s) 4 cores 305 ms

  8. for comprehensions surnames.par.flatMap { s => names.par .filter(n => s endsWith n) .map(n => (n, s)) }

  9. for comprehensionsnested parallelized bulk operations surnames.par.flatMap { s => names.par .filter(n => s endsWith n) .map(n => (n, s)) }

  10. Nested parallelism

  11. Nested parallelismparallel within parallel surnames.par.flatMap { s => surnameToCollection(s) // may invoke parallel ops } composition

  12. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ...

  13. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc

  14. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield recursive algorithms

  15. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c

  16. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c

  17. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, Array(""))

  18. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: Seq[String]): Seq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, Array("")) 1545 ms

  19. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: ParSeq[String]): ParSeq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, ParArray(""))

  20. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: ParSeq[String]): ParSeq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, ParArray("")) 1 core 1575 ms

  21. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: ParSeq[String]): ParSeq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, ParArray("")) 2 cores 809 ms

  22. Nested parallelismgoing recursive def vowel(c: Char): Boolean = ... def gen(n: Int, acc: ParSeq[String]): ParSeq[String] = if (n == 0) acc else for (s <- gen(n - 1, acc); c <- 'a' to 'z') yield if (s.length == 0) s + c else if (vowel(s.last) && !vowel(c)) s + c else if (!vowel(s.last) && vowel(c)) s + c else s gen(5, ParArray("")) 4 cores 530 ms

  23. So, I just use par and I’m home free?

  24. How to think parallel

  25. Character countuse case for foldLeft val txt: String = ... txt.foldLeft(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 }

  26. Character countuse case for foldLeft txt.foldLeft(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } going left to right - not parallelizable! 2 4 3 5 1 0 6 A B C D E F _ + 1

  27. Character countuse case for foldLeft txt.foldLeft(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } going left to right – not really necessary 6 _ + _ 2 1 0 3 A B C 0 2 3 1 D E F _ + 1 _ + 1

  28. Character countin parallel txt.fold(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 }

  29. Character countin parallel txt.fold(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } : (Int, Char) => Int 2 1 1 3 A B C 3 2 1 1 A B C _ + 1

  30. Character countfold not applicable txt.fold(0) { case (a, ‘ ‘) => a case (a, c) => a + 1 } ! (Int, Int) => Int 3 3 _ + _ 3 2 1 3 A B C 3 2 1 3 A B C

  31. Character countuse case for aggregate txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _)

  32. Character countuse case for aggregate txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _) 3 3 _ + _ 3 1 1 2 A B C 3 2 1 3 A B C _ + 1

  33. Character countuse case for aggregate txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _) aggregation  element 3 3 _ + _ 1 2 3 1 A B B C 3 2 3 1 A B C _ + 1

  34. Character countuse case for aggregate txt.aggregate(0)({ case (a, ‘ ‘) => a case (a, c) => a + 1 }, _ + _) aggregation  aggregation aggregation  element 3 3 _ + _ 1 2 3 1 A B B C 3 2 3 1 A B C _ + 1

  35. Word countanother use case for foldLeft txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) }

  36. Word countinitial accumulation txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } 0 words so far last character was a space “Folding me softly.”

  37. Word counta space txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } last seen character is a space “Folding me softly.”

  38. Word counta non space txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } last seen character was a space – a new word “Folding me softly.”

  39. Word counta non space txt.foldLeft((0, true)) { case ((wc, _), ' ') => (wc, true) case ((wc, true), x) => (wc + 1, false) case ((wc, false), x) => (wc, false) } last seen character wasn’t a space – no new word “Folding me softly.”

  40. Word countin parallel P1 P2 “Folding me “ “softly.“

  41. Word countin parallel P1 P2 “Folding me “ “softly.“  wc = 2; rs = 1 wc = 1; ls = 0

  42. Word countin parallel P1 P2 “Folding me “ “softly.“  wc = 2; rs = 1 wc = 1; ls = 0 wc = 3

  43. Word countmust assume arbitrary partitions P1 P2 “Foldin“ “g me softly.“  wc = 1; rs = 0 wc = 3; ls = 0

  44. Word count must assume arbitrary partitions P1 P2 “Foldin“ “g me softly.“  wc = 1; rs = 0 wc = 3; ls = 0 wc = 3

  45. Word countinitial aggregation txt.par.aggregate((0, 0, 0))

  46. Word countinitial aggregation txt.par.aggregate((0, 0, 0)) # spaces on the left #words # spaces on the right

  47. Word countinitial aggregation txt.par.aggregate((0, 0, 0)) # spaces on the left #words # spaces on the right ””

  48. Word countaggregation  aggregation ... }, { case ((0, 0, 0), res) => res case (res, (0, 0, 0)) => res  ““ “softly.“  “Folding me“ ““

More Related