dataflow systems extensions for graphs recursion n.
Skip this Video
Loading SlideShow in 5 Seconds..
Dataflow Systems Extensions for Graphs Recursion PowerPoint Presentation
Download Presentation
Dataflow Systems Extensions for Graphs Recursion

Loading in 2 Seconds...

play fullscreen
1 / 37

Dataflow Systems Extensions for Graphs Recursion - PowerPoint PPT Presentation

  • Uploaded on

Extensions of MapReduce. Dataflow Systems Extensions for Graphs Recursion. Jeffrey D. Ullman Stanford University. Dataflow Systems. Arbitrary Acyclic Flow Among Tasks Preserving Fault Tolerance The Blocking Property. Generalization of MapReduce.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Dataflow Systems Extensions for Graphs Recursion' - audrey-leach

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
dataflow systems

Dataflow Systems

Arbitrary Acyclic Flow Among Tasks

Preserving Fault Tolerance

The Blocking Property

generalization of mapreduce
Generalization of MapReduce
  • MapReduceuses only two functions (Map and Reduce).
    • Each is implemented by a rank of tasks.
    • Data flows from Map tasks to Reduce tasks only.
generalization 2
Generalization – (2)
  • Natural generalization is to allow any number of functions, connected in an acyclic network.
  • Each function implemented by tasks that feed tasks of successor function(s).
  • Key fault-tolerance (blocking) property: tasks produce all their output at the end.
  • Important point: Map tasks never deliver their output until completed.
    • Thus, we can restart a Map task that failed without fear that a Reduce task has already used some output of the failed Map task.
many implementations
Many Implementations
  • Clustera – University of Wisconsin.
  • Hyracks – Univ. of California/Irvine.
  • Dryad/DryadLINQ – Microsoft.
  • Nephele/PACT – T. U. Berlin.
  • BOOM – Berkeley.
  • epiC – N. U. Singapore.
example join aggregation
Example: Join + Aggregation
  • Relations D(emp, dept) and S(emp, salary).
  • Compute the sum of the salaries for each department.
  • D JOIN S computed by MapReduce.
    • But each Reduce task can also group its emp-dept-salarytuples by dept and sum the salaries.
  • A Third function is needed to take the dept-SUM(salary) pairs from each Reduce task, organize them by dept, and compute the final sum for each department.
3 layer dataflow


Group +



Join +









3-Layer Dataflow







Transitive-Closure Example

Fault-Tolerance Problem

Endgame Problem

Some Systems and Approaches

applications requiring recursion
Applications Requiring Recursion
  • PageRank, the original map-reduce application is really a recursion implemented by many rounds of map-reduce.
  • Analysis of social networks.
  • Many machine-learning algorithms, e.g., gradient descent.
  • PDE’s.
transitive closure

Nonlinear. Takes

log n rounds on an

n-node graph.

(Right) Linear. Takes

n rounds on an n-node


Transitive Closure
  • Many recursive applications involving large data are similar to transitive closure :

Path(X,Y) :- Arc(X,Y)

Path(X,Y) :- Path(X,Z) & Path(Z,Y)

Path(X,Y) :- Arc(X,Y)

Path(X,Y) :- Arc(X,Z) & Path(Z,Y)

implementing tc on a cluster
Implementing TC on a Cluster
  • Use k tasks.
  • Nonlinear recursion used here.
  • Hash function h sends each node of the graph to one of the k tasks.
  • Task i receives and stores Path(a,b) if either h(a) = i or h(b) = i, or both.
  • Task i must join Path(a,c) with Path(c,b) if h(c) = i.
tc on a cluster basis
TC on a Cluster – Basis
  • Data is stored as relation Arc(a,b).
  • “Map” tasks read chunks of the Arc relation and send each tuple Arc(a,b) to recursive tasks h(a) and h(b).
    • Treated as if it were tuple Path(a,b).
    • If h(a) = h(b), only one task receives.
tc on a cluster recursive tasks

Send Path(a,c) to

tasks h(a) and h(c);

send Path(d,b) to

tasks h(d) and h(b)



Task i



if new.



Look up

Path(b,c) and/or

Path(d,a) for

any c and d

TC on a Cluster – Recursive Tasks
big problem managing failure
Big Problem: Managing Failure
  • MapReducedepends on the blocking property.
  • Only then can you restart a failed task without restarting the whole job.
  • But any recursive task has to deliver some output and later get more input.
haloop u washington
HaLoop (U. Washington)
  • Iterates Hadoop, once for each round of the recursion.
    • Uses Hadoop blocking-based fault tolerance.
  • Similar idea: Twister (U. Indiana).
  • HaLooptries to run each task in round iat a compute node where it can find its needed output from round i– 1.
  • Also partitions and stores locally a file that is used at each round.
    • Example: Arc in Path(X,Y) :- Arc(X,Z) & Path(Z,Y)
pregel google
Pregel (Google)
  • Views all computation as a recursion on some graph.
  • Nodes send messages to one another.
    • Messages bunched into supersteps, where each node processes all data received.
    • Sending individual messages would result in far too much overhead.
  • Checkpoint all compute nodes after some fixed number of supersteps.
  • On failure, rolls all tasks back to previous checkpoint.
example shortest paths via pregel

Is this the

shortest path from

M I know about?

If so …

I found a path

from node M to

you of length L

I found a path

from node M to

you of length L+5

I found a path

from node M to

you of length L+6

I found a path

from node M to

you of length L+3

Example: Shortest Paths Via Pregel



table of



to N




other graph oriented systems
Other Graph-Oriented Systems
  • Giraph: open-source Pregel.
  • GraphLab: similar system that deals more effectively with nodes of high degree.
    • Will split the work for such a graph node among several compute nodes.
using idempotence
Using Idempotence
  • Some recursive applications allow restart of tasks even if they have produced some output.
  • Example: TC is idempotent; you can send a task a duplicate Path fact without altering the result.
    • But if you were countingpaths, the answer would be wrong.
big problem the endgame
Big Problem: The Endgame
  • Some recursions, like TC, take a large number of rounds, but the number of new discoveries in later rounds drops.
    • T. Vassilakis: searches forward on the Web graph can take hundreds of rounds.
  • Problem: in a cluster, transmitting small files carries much overhead.
approach merge tasks
Approach: Merge Tasks
  • Decide when to migrate tasks to fewer compute nodes.
  • Data for several tasks at the same node are combined into a single file and distributed at the receiving end.
  • Downside: old tasks have a lot of state to move.
    • Example: “paths seen so far.”
approach modify algorithms
Approach: Modify Algorithms
  • Nonlinear recursions can terminate in many fewer steps than equivalent linear recursions.
    • Avoids the endgame problem.
  • Example: TC.
    • O(n) rounds on n-node graph for linear.
    • O(log n) rounds for nonlinear.
advantage of linear tc
Advantage of Linear TC
  • The communication cost (= sum of input sizes of all tasks) for executing linear TC is generally lower than that for nonlinear TC.
  • Why? Each path is discovered only once (unique-decomposition property).
    • Note: distinct paths between the same endpoints may each be discovered.
smart tc
Smart TC
  • (Valduriez-Boral, Ioannides) Construct a path from two paths:
    • The first has a length that is a power of 2.
    • The second is no longer than the first.
other nonlinear tc algorithms
Other Nonlinear TC Algorithms
  • You can have the unique-decomposition property with many variants of nonlinear TC.
  • Example: Balanceconstructs paths from two equal-length paths.
    • Favor first path when length is odd.
incomparability of tc algorithms
Incomparability of TC Algorithms
  • On different graphs, any of the unique-decomposition algorithms – left-linear, right-linear, smart, balanced – could have the lowest data-volume cost.
  • Other unique-decomposition algorithms are possible and also could win.
extension beyond tc
Extension Beyond TC
  • Can you avoid the endgame problem by converting any linear recursion into an equivalent nonlinear recursion that requires logarithmic rounds?
  • Answer: Not always, without increasing arity and data size.
positive points
Positive Points
  • (Agarwal, Jagadish, Ness) All linear Datalog recursions reduce to TC.
  • Right-linear chain-rule Datalog programs can be replaced by nonlinear recursions with the same arity, logarithmic rounds, and the unique-decomposition property.

Each subgoal shares variables

only with the next, in a circular

sense that includes the head.

example alternating color paths
Example: Alternating-Color Paths

P(X,Y) :- Blue(X,Y)

P(X,Y) :- Blue(X,Z) & Q(Z,Y)

Q(X,Y) :- Red(X,Z) & P(Z,Y)

the case of reachability
The Case of Reachability

Reach(X) :- Source(X)

Reach(X) :- Reach(Y) & Arc(Y,X)

  • Takes linear rounds as stated.
  • Can compute nonlinear TC to get Reach in O(log n) rounds.
  • But, then you compute O(n2) facts instead of O(n) facts on an n-node graph.
reachability 2
Reachability – (2)
  • Theorem: If you compute Reach using only unary recursive predicates, then it must take (n) rounds on a graph of n nodes.
    • Proof uses the ideas of Afrati, Cosmodakis, and Yannakakis from a generation ago.
summary recursion
Summary: Recursion
  • Key problems are “endgame” and nonblocking nature of recursive tasks.
  • In some applications, endgame problem can be handled by using a nonlinear recursion that requires O(log n) rounds and has the unique-decomposition property.
summary research questions
Summary: Research Questions
  • How do you best support fault tolerance when tasks are nonblocking?
  • How do you manage tasks when the endgame problem cannot be avoided?
  • When can you replace linear recursion with nonlinear recursion requiring many fewer rounds, (roughly) the same communication cost, and (roughly) the same number of facts discovered?