Reef retainable evaluator execution framework
This presentation is the property of its rightful owner.
Sponsored Links
1 / 16

REEF Retainable Evaluator Execution Framework PowerPoint PPT Presentation


  • 73 Views
  • Uploaded on
  • Presentation posted in: General

REEF Retainable Evaluator Execution Framework. Rusty Sears Microsoft Cloud and Information Services Lab (CISL). Resource Managers. True multi-tenancy … Unified realtime -batch workflows In-situ processing Utilization: one cluster for scientists and production

Download Presentation

REEF Retainable Evaluator Execution Framework

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Reef retainable evaluator execution framework

REEFRetainable Evaluator Execution Framework

Rusty Sears

Microsoft Cloud and Information Services Lab (CISL)


Reef retainable evaluator execution framework

Resource Managers

True multi-tenancy…

Unified realtime-batch workflows

In-situ processing

Utilization: one cluster for scientists and production

…but, only for sophisticated apps


Reef retainable evaluator execution framework

Resource Managers

True multi-tenancy…

Unified realtime-batch workflows

In-situ processing

Utilization: one cluster for scientists and production

…but, only for sophisticated apps

Fault tolerance

Pre-emption

Elasticity


Reef retainable evaluator execution framework

Running SQL / MapReduce

Checkpointing

Fault tolerance

Elasticity


Reef retainable evaluator execution framework

Machine learning

Checkpointing

Fault tolerance

Elasticity

Iterative computations


Reef retainable evaluator execution framework

Graph processing

Checkpointing

Fault tolerance

Elasticity

Iterative computations

Low latency communication


Reef retainable evaluator execution framework

Problems with silos

Tedious: users write code to dump + load data at each step

Slow: Data unnecessarily written to disk, read back (and re-parsed) at each step

Hard to build: Each duplicates the same mechanisms under the hood


Reef retainable evaluator execution framework

REEF’s goals

Support YARN versions of new (and existing) scalable data pipelines.

Allow them to be transparently composed.

Move redundant tooling and plumbing into shared libraries.


Reef retainable evaluator execution framework

+ 3

REEF control flow

Yarn ( ) handles resource management (security, quotas, priorities)

Per-job Drivers ( ) request resources, coordinate computations, and handle faults, preemption, etc…

REEF Evaluators ( ) hold hardware resources, allowing multiple Activities

( , , , , , , etc…) to use the same cached state.


Reef retainable evaluator execution framework

M

M

Retaining Evaluators

R

R

R

R

Handover of pre-partitioned and parsed data between frameworks

Iterative computation

Interactive queries

$…


Reef retainable evaluator execution framework

Wake: Events + I/O

Thread per connection / file doesn’t scale

Provide static subset of Rx

→ static checking of event flows

→ aggressive JVM event inlining

Latency, throughput profiler


Reef retainable evaluator execution framework

A

B

A

B

REEF data plane

B

Fault-tolerant async communication

Group communication / shuffle

Low-latency communication

Storage, checkpointing, preemption


Reef retainable evaluator execution framework

Tang: Config Manager

Configuring distributed systems is hard

So is reasoning about event flows

Tang performs static and dynamic checks to help ease the pain


Reef retainable evaluator execution framework

cmd = ‘ls’

Command = ‘ls’

ShellActivity

Tang: Config Manager

Error:

container-4872364523847-02.stderr:

NullPointerExceptionat:

java…eval():1234

ShellActivity.helper():546

ShellActivity.onNext():789

YarnEvaluator.onNext():12

Evaluator

Activity

YarnEvaluator

Evaluator

Configuring distributed systems is hard

So is reasoning about event flows

Tang performs static and dynamic checks to help ease the pain

Error:

Unknown parameter “Command”

Missing required parameter “cmd”

Error:

Required instanceof Evaluator

Got ShellActivity

#!


Reef retainable evaluator execution framework

Open source release soon!

Apache 2.0 licensed

Java support first, C# to follow

Interested in early access? Ping us!


Reef retainable evaluator execution framework

  • [email protected]


  • Login