1 / 36

Sparrow

Sparrow. Kay Ousterhout, Patrick Wendell, Matei Zaharia , Ion Stoica. Distributed Low-Latency Spark Scheduling. Outline. The Spark scheduling bottleneck Sparrow’s fully distributed, fault-tolerant technique Sparrow’s near-optimal performance. Spark Today. User 1. Worker. Spark Context.

kipling
Download Presentation

Sparrow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Sparrow Kay Ousterhout, Patrick Wendell, MateiZaharia, Ion Stoica Distributed Low-Latency Spark Scheduling

  2. Outline The Spark scheduling bottleneck Sparrow’s fully distributed, fault-tolerant technique Sparrow’s near-optimal performance

  3. Spark Today User 1 Worker Spark Context Worker Worker Query Compilation User 2 Worker Storage Worker Scheduling User 3 Worker

  4. Spark Today User 1 Worker Spark Context Worker Worker Query Compilation User 2 Worker Storage Worker Scheduling User 3 Worker

  5. Job Latencies Rapidly Decreasing 2012: Impala query 2010: Dremel Query 2010: In-memory Spark query 2004: MapReduce batch job 2009: Hive query 2013: Spark streaming 10 sec. 10 min. 100 ms 1 ms

  6. Job latencies rapidly decreasing

  7. Job latencies rapidly decreasing + Spark deployments growing in size Scheduling bottleneck!

  8. Spark scheduler throughput: 1500 tasks / second Cluster size (# 16-core machines) Task Duration 1000 10 second 100 1 second 10 100 ms

  9. Optimizing the Spark Scheduler 0.8: Monitoring code moved off critical path 0.8.1: Result deserialization moved off critical path Future improvements may yield 2-3x higher throughput

  10. Is the scheduler the bottleneck in my cluster?

  11. Task launch Worker Worker Worker Cluster Scheduler Worker Worker Worker Task completion

  12. Task launch Worker Worker Worker Cluster Scheduler Worker Worker Worker Task completion

  13. Task launch Worker Worker Worker Cluster Scheduler Worker Scheduler delay Worker Worker Task completion

  14. Spark Today User 1 Worker Spark Context Worker Worker Query Compilation User 2 Worker Storage Worker Scheduling User 3 Worker

  15. Future Spark User 1 Worker Scheduler Query compilation Benefits: High throughput Fault tolerance Worker Worker User 2 Scheduler Query compilation Worker Worker User 3 Scheduler Query compilation Worker

  16. Future Spark User 1 Worker Scheduler Query compilation Worker Storage: Tachyon Worker User 2 Scheduler Query compilation Worker Worker User 3 Scheduler Query compilation Worker

  17. Scheduling with Sparrow Worker Scheduler Worker Stage Scheduler Worker Worker Scheduler Worker Scheduler Worker

  18. Batch Sampling 4 probes (d = 2) Worker Scheduler Worker Stage Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of 2m workers

  19. 80 ms Queue length poor predictor of wait time 155 ms Worker Worker 530 ms Poor performance on heterogeneous workloads

  20. Late Binding 4 probes (d = 2) Worker Scheduler Worker Stage Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of dmworkers

  21. Late Binding 4 probes (d = 2) Worker Scheduler Worker Stage Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of dmworkers

  22. Late Binding Worker requests task Worker Scheduler Worker Stage Scheduler Worker Worker Scheduler Worker Scheduler Worker Place m tasks on the least loaded of dmworkers

  23. What about constraints?

  24. Per-Task Constraints Probe separately for each task Worker Scheduler Worker Stage Scheduler Worker Worker Scheduler Worker Scheduler Worker

  25. Technique Recap Worker Scheduler Batch sampling + Late binding + Constraints Worker Scheduler Worker Worker Scheduler Worker Scheduler Worker

  26. How well does Sparrow perform?

  27. How does Sparrow compare to Spark’s native scheduler? 100 16-core EC2 nodes, 10 tasks/job, 10 schedulers, 80% load

  28. TPC-H Queries: Background TPC-H: Common benchmark for analytics workloads Shark: SQL execution engine Spark Sparrow

  29. TPC-H Queries Percentiles 100 16-core EC2 nodes, 10 schedulers, 80% load 95 Within 12% of ideal Median queuing delay of 9ms 75 50 25 5

  30. Policy Enforcement Priorities Serve queues based on strict priorities Fair Shares Serve queues using weighted fair queuing Worker Worker High Priority User A (75%) User B (25%) Low Priority

  31. Weighted Fair Sharing

  32. Fault Tolerance Timeout: 100ms Failover: 5ms Re-launch queries: 15ms ✗ Scheduler 1 Spark Client 1 Spark Client 2 Scheduler 2

  33. Making Sparrow feature-complete Interfacing with UI Delay scheduling Speculation

  34. www.github.com/radlab/sparrow (1) Diagnosing a Spark scheduling bottleneck Worker Scheduler Worker Scheduler Worker (2) Distributed, fault-tolerant scheduling with Sparrow Worker Scheduler Worker Scheduler Worker

More Related