Reef retainable evaluator execution framework
Download
1 / 16

REEF Retainable Evaluator Execution Framework - PowerPoint PPT Presentation


  • 102 Views
  • Uploaded on

REEF Retainable Evaluator Execution Framework. Rusty Sears Microsoft Cloud and Information Services Lab (CISL). Resource Managers. True multi-tenancy … Unified realtime -batch workflows In-situ processing Utilization: one cluster for scientists and production

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' REEF Retainable Evaluator Execution Framework' - gilon


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Reef retainable evaluator execution framework

REEFRetainable Evaluator Execution Framework

Rusty Sears

Microsoft Cloud and Information Services Lab (CISL)


Resource Managers

True multi-tenancy…

Unified realtime-batch workflows

In-situ processing

Utilization: one cluster for scientists and production

…but, only for sophisticated apps


Resource Managers

True multi-tenancy…

Unified realtime-batch workflows

In-situ processing

Utilization: one cluster for scientists and production

…but, only for sophisticated apps

Fault tolerance

Pre-emption

Elasticity


Running SQL / MapReduce

Checkpointing

Fault tolerance

Elasticity


Machine learning

Checkpointing

Fault tolerance

Elasticity

Iterative computations


Graph processing

Checkpointing

Fault tolerance

Elasticity

Iterative computations

Low latency communication


Problems with silos

Tedious: users write code to dump + load data at each step

Slow: Data unnecessarily written to disk, read back (and re-parsed) at each step

Hard to build: Each duplicates the same mechanisms under the hood


REEF’s goals

Support YARN versions of new (and existing) scalable data pipelines.

Allow them to be transparently composed.

Move redundant tooling and plumbing into shared libraries.


+ 3

REEF control flow

Yarn ( ) handles resource management (security, quotas, priorities)

Per-job Drivers ( ) request resources, coordinate computations, and handle faults, preemption, etc…

REEF Evaluators ( ) hold hardware resources, allowing multiple Activities

( , , , , , , etc…) to use the same cached state.


M

M

Retaining Evaluators

R

R

R

R

Handover of pre-partitioned and parsed data between frameworks

Iterative computation

Interactive queries

$…


Wake: Events + I/O

Thread per connection / file doesn’t scale

Provide static subset of Rx

→ static checking of event flows

→ aggressive JVM event inlining

Latency, throughput profiler


A

B

A

B

REEF data plane

B

Fault-tolerant async communication

Group communication / shuffle

Low-latency communication

Storage, checkpointing, preemption


Tang: Config Manager

Configuring distributed systems is hard

So is reasoning about event flows

Tang performs static and dynamic checks to help ease the pain


cmd = ‘ls’

Command = ‘ls’

ShellActivity

Tang: Config Manager

Error:

container-4872364523847-02.stderr:

NullPointerExceptionat:

java…eval():1234

ShellActivity.helper():546

ShellActivity.onNext():789

YarnEvaluator.onNext():12

Evaluator

Activity

YarnEvaluator

Evaluator

Configuring distributed systems is hard

So is reasoning about event flows

Tang performs static and dynamic checks to help ease the pain

Error:

Unknown parameter “Command”

Missing required parameter “cmd”

Error:

Required instanceof Evaluator

Got ShellActivity

#!


Open source release soon!

Apache 2.0 licensed

Java support first, C# to follow

Interested in early access? Ping us!



ad