1 / 25

Coflow

Coflow. A Networking Abstraction For Cluster Applications. Mosharaf Chowdhury Ion Stoica. UC Berkeley. Cluster Applications. Multi-Stage Data Flows Computation interleaved with communication Computation Distributed Runs on many machines Communication Structured Between machine groups.

Download Presentation

Coflow

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Coflow A Networking Abstraction For Cluster Applications Mosharaf Chowdhury Ion Stoica UC Berkeley

  2. Cluster Applications Multi-Stage Data Flows • Computation interleaved with communication Computation • Distributed • Runs on many machines Communication • Structured • Between machine groups Driver

  3. Communication Abstraction A Flow • Sequence of packets • Independent • Often the unit for network scheduling, traffic engineering, load balancing etc. Multiple Parallel Flows • Independent • Yet, semantically bound • Shared objective Driver Minimize Completion Time

  4. Coflow ‘ A collection of flows between two groups of machines that are bound together by application-specific semantics Captures • Structure • Shared Objective • Semantics

  5. We Want To… Better schedule the network • Intra-coflow • Inter-coflow Write the communication layer of a new application • Without reinventing the wheel Add unsupported coflows to an application, or Replace an existing coflow implementation • Independent of applications

  6. Cluster Applications Coflow API The Network (Physically or Logically Centralized Controller)

  7. Coflow API Goals • Separate intent from mechanisms • Convey application-specific semantics to the network

  8. Coflow API Jobfinishes terminate(handle) Shuffle finishes get(handle, id) content Driver put(handle, id, content) MapReduce create(SHUFFLE) handle

  9. Choice of algorithms • Default • WSS1 Choice of mechanism • App vs. Network layer • Pull vs. Push Coflow Flexibility reducers shuffle mappers 1. Orchestra, SIGCOMM’2011

  10. @driver bcreate(BCAST) … put(b, id, content) … terminate(b) Coflow Flexibility reducers shuffle mappers @mapper get(b, id) … broadcast driver (JobTracker)

  11. @driver bcreate(BCAST) screate(SHUFFLE, ord=[b ~> s]) put(b, id, content) … terminate(b) terminate(s) Coflow Flexibility reducers shuffle mappers @mapper get(b, id) put(s, ids1) … broadcast driver (JobTracker)

  12. Throughput-Sensitive Applications After 2 seconds Minimize Completion Time

  13. Throughput-Sensitive Applications After 4 seconds After 7 seconds After 2 seconds Minimize Completion Time

  14. Throughput-Sensitive Applications After 7 seconds Free up resources without hurting application-perceived communication time After 2 seconds Minimize Completion Time

  15. Latency-Sensitive Applications HotNets 2012 Top-level Aggregator Mid-level Aggregators Workers

  16. Latency-Sensitive Applications HotNets-XI: Home Page conferences.sigcomm.org/hotnets/2012/ The Eleventh ACM Workshop on Hot Topics in Networks (HotNets-XI) will bring together people with interest in computer networks to engage in a lively debate ... HotNets Workshop | acm sigcomm www.sigcomm.org/events/hotnets-workshop The Workshop on Hot Topics in Networks (HotNets) was created in 2002 to discuss early-stage, creative ... HotNets-XI, Seattle, WA area, October 29-30, 2012. HotNets-XI: Call for Papers conferences.sigcomm.org/hotnets/2012/cfp.shtml The Eleventh ACM Workshop on Hot Topics in Networks (HotNets-XI) will bring together researchers in computer networks and systems to engage in a lively ... Coflow accepted at HotNets'2012 www.mosharaf.com/blog/2012/09/.../coflow-accepted-at-hotnets201... Sep 13, 2012 – Update: Coflow camera-ready is available online! Tell us what you think! Our position paper to address the lack of a networking abstraction for ... HotNets 2012 Top-level Aggregator Meet Deadline1,2 Mid-level Aggregators Meet Deadline1,2 Workers Limit impact to as few requests as possible 1. D3, SIGCOMM’2011 2. PDQ, SIGCOMM’2012

  17. One More Thing… • Critical Path Scheduling • OpenTCP • Structured Streams • …

  18. Coflow • A semantically-bound collection of flows • Conveys application intent to the network • Allows better management of network resources • Provides greater flexibility in designing applications Mosharaf Chowdhuryhttp://www.mosharaf.com/ UC Berkeley

  19. Critical Path Scheduling Communication of a cluster application is represented by a partially-ordered set of coflows Network allocation takes place among these partially-ordered sets of coflows A A S S B S S S

  20. Coflow API

  21. Throughput-Sensitive Applications Job finishes Reduce Stage Shuffle finishes Local shuffle finishes Local shuffle finishes Minimize Completion Time1 Map Stage MapReduce Framework Data Flow 1. Orchestra, SIGCOMM’2011

  22. reducers reducers Coflow Resource Allocation 1. Weights [Across Apps] Weighted sharing between coflows @driver shuffle1 create(SHUFFLE, weight=1) shuffle2 create(SHUFFLE, weight=2) … shuffle2 shuffle1 mappers mappers Job 2 Job 1

  23. reducers reducers Strict priorities @driver shuffle1 create(SHUFFLE, pri=3) shuffle2 create(SHUFFLE, pri=5) … Coflow Resource Allocation 2. Priorities [Across Apps] shuffle2 shuffle1 mappers mappers Job 2 Job 1

  24. reducers reducers Coflow Resource Allocation 3. Dependencies [Within Apps] finishes_before (~>) @driver bcreate(BCAST) shuffle2create(SHUFFLE, ord=[b ~> shuffle2]) agg create(AGGR, ord=[shuffle2 ~> agg]) shuffle2 shuffle1 aggregation(agg) mappers mappers broadcast (b) driver Job 2 Job 1

  25. Coflow Resource Allocation Communication of a cluster application is represented by a partially-ordered set of coflows Network allocation takes place among these partially-ordered sets of coflows A A S S S B S S

More Related