Extensible Distributed Tracing from Kernels to Clusters - PowerPoint PPT Presentation

Fay

Extensible Distributed Tracing from Kernels to Clusters

Úlfar Erlingsson, Google Inc.

Marcus Peinado, Microsoft Research

Simon Peter, Systems Group, ETH Zurich

MihaiBudiu, Microsoft Research

• We could know what our clusters were doing?
• We could ask any question, … easily, using one simple-to-use system.
• We could collect answers extremely efficiently … so cheaply we may even ask continuously.

• Applying data-mining to cluster tracing
• Bag of words technique
• Compare documents w/o structural knowledge
• N-dimensional feature vectors
• K-means clustering
• Can apply to clusters, too!

• Automatically categorize cluster behavior, based on system call activity

• Automatically categorize cluster behavior, based on system call activity
• Without measurable overhead on the execution
• Without any special Fay data-mining support

var kernelFunctionFrequencyVectors =

cluster.Function(kernel, “syscalls!*”)

.Where(evt => evt.time < Now.AddMinutes(3))

.Select(evt => new { Machine = fay.MachineID(),

Interval = evt.Cycles / CPS,

Function = evt.CallerAddr })

.GroupBy(evt => evt,

(k,g) => new { key = k, count = g.Count() });

Vector Nearest(Vector pt, Vectors centers) {

var near = centers.First();

foreach (var c in centers)

if (Norm(pt – c) < Norm(pt – near))

near = c;

return near;

}

Vectors OneKMeansStep(Vectors vs, Vectors cs) {

return vs.GroupBy(v => Nearest(v, cs))

.Select(g => g.Aggregate((x,y)

=> x+y)/g.Count());

}

Vectors KMeans(Vectors vs, Vectors cs, int K) {

for (int i=0; i < K; ++i)

cs = OneKMeansStep(vs, cs);

return cs;

}

var kernelFunctionFrequencyVectors =

cluster.Function(kernel, “syscalls!*”)

.Where(evt => evt.time < Now.AddMinutes(3))

.Select(evt => new { Machine = fay.MachineID(),

Interval = evt.Cycles / CPS,

Function = evt.CallerAddr })

.GroupBy(evt => evt,

(k,g) => new { key = k, count = g.Count() });

• Could’ve built a specialized tool for this
• Automatic categorization of behavior (Fmeter)
• Fay is general, but can efficiently do
• Tracing across abstractions, systems (Magpie)
• Predicated and windowed tracing (Streams)
• Probabilistic tracing (Chopstix)
• Flight recorders, performance counters, …

Fay: Flexible monitoring of distributed executions

• Can be applied to existing, live Windows servers
• Single query specifies both tracing & analysis
• Easy to write & enables automatic optimizations
• Pervasively data-parallel,scalable processing
• Same model within machines & across clusters
• Inline, safe machine-code at tracepoints
• Allows us to do computation right at data source

var kernelFunctionFrequencyVectors =

cluster.Function(kernel, “*”)

.Where(evt => evt.time < Now.AddMinutes(3))

.Select(evt => new { Machine = fay.MachineID(),

Interval = evt.Cycles / CPS,

Function = evt.CallerAddr})

.GroupBy(evt => evt,

(k,g) => new { key = k, count = g.Count() });

var kernelFunctionFrequencyVectors =

cluster.Function(kernel, “*”)

.Where(evt => evt.time < Now.AddMinutes(3))

.Select(evt => new { Machine = MachineID(),

Interval = w.Cycles / CPS,

Function = w.CallerAddr})

.GroupBy(evt => evt,

(k,g) => new { key = k, count = g.Count() });

Vector Nearest(Vector pt, Vectors centers) {

var near = centers.First();

foreach (var c in centers)

if (|pt – c| < |pt – near|)

near = c;

return near;

}

Vector Nearest(Vector pt, Vectors centers) {

var near = centers.First();

foreach (var c in centers)

if (Norm(pt – c) < Norm(pt – near))

near = c;

return near;

}

Vectors OneKMeansStep(Vectors vs, Vectors cs) {

return vs.GroupBy(v => Nearest(v, cs))

.Select(g => g.Aggregate((x,y)

=> x+y)/g.Count());

}

Vectors KMeans(Vectors vs, Vectors cs, int K) {

for (int i=0; i < K; ++i)

cs = OneKMeansStep(vs, cs);

return cs;

}

Vectors OneKMeansStep(Vectors vs, Vectors cs) {

return vs.GroupBy(v => Nearest(v, cs))

.Select(g => g.Aggregate((x,y)

=> x+y)/g.Count());

}

Vectors KMeans(Vectors vs, Vectors cs, int K) {

for (int i=0; i < K; ++i)

cs = OneKMeansStep(vs, cs);

return cs;

}

• View trace query as distributed computation
• Use cluster for analysis

• System call trace events
• Fay does early aggregation & data reduction
• Fay knows what’s needed for later analysis

• System call trace events
• Fay does early aggregation & data reduction
• K-Means analysis
• Fay builds an efficient processing plan from query

• Early aggregation
• Inline, in OS kernel
• Reduce dataflow & kernel/user transitions
• Data-parallel per each core/thread

K-Means: System calls

K-Means: Clustering

• Collect data first (on disk)
• Reduce later
• Inefficient, can suffer data overload

K-Means: System calls

K-Means: Clustering

• First log all data (a deluge)
• Process later (centrally)
• Compose tools via scripting

Fay: Flexible monitoring of distributed executions

• Single query specifies both tracing & analysis
• Pervasively data-parallel,scalable processing

• A variant of XFI used for safety [OSDI’06]
• Works well in the kernel or any address space
• Can safely use existing stacks, etc.
• Instead of language interpreter (DTrace)
• Arbitrary, efficient, statefulcomputation
• Probes can access thread-local/global state
• Probes can try to read any address
• I/O registers are protected

Fay: Flexible monitoring of distributed executions

• Single query specifies both tracing & analysis
• Pervasively data-parallel,scalable processing
• Inline, safe machine-code at tracepoints

• Fay runtime on each machine
• Fay module in each traced address space
• Tracepoints at hotpatched function boundary

Tracing Runtime

query

Createprobe

ETW

User-Space

Kernel

Target

Fay

Probe

XFI

Hotpatching

200 cycles

Module with a traced function Foo

Caller:

...

e8ab62ffff call Foo

...

ff1508e70600 call[Dispatcher]

Foo: ebf8 jmp Foo-6

cccccc

Foo2: 57 push rdi

...

c3 ret

• Replace 1stopcode of functions

Module with a traced function Foo

Fay platform module

Dispatcher:

t = lookup(return_addr)

...

call t.entry_probes

...

call t.Foo2_trampoline

...

call t.return_probes

...

return /* to after call Foo */

Caller:

...

e8ab62ffff call Foo

...

ff1508e70600 call[Dispatcher]

Foo: ebf8 jmp Foo-6

cccccc

Foo2: 57 push rdi

...

c3 ret

• Replace 1stopcode of functions
• Fay dispatcher called via trampoline

Module with a traced function Foo

Fay platform module

Fay probes

Dispatcher:

t = lookup(return_addr)

...

call t.entry_probes

...

call t.Foo2_trampoline

...

call t.return_probes

...

return /* to after call Foo */

Caller:

...

e8ab62ffff call Foo

...

ff1508e70600 call[Dispatcher]

Foo: ebf8 jmp Foo-6

cccccc

Foo2: 57 push rdi

...

c3 ret

PF3

XFI

PF4

PF5

XFI

XFI

• Replace 1stopcode of functions
• Fay dispatcher called via trampoline
• Fay calls the function, and entry & exit probes

• Fay adds 220 to 430 cycles per traced function
• Fay adds 180% CPU to trace all kernel functions
• Both approx 10x faster than Dtrace, SystemTap

Slowdown (x)

Null-probe overhead

Cycles

• Fay tracing memory allocations, in a loop:
• Ran workload on a 128-node, 1024-core cluster
• Spread work over 128 to 1,280,000 threads
• 100% CPU utilization
• Fay overhead was 1% to 11% (mean 7.8%)

• Details of query-plan optimizations
• Case studies of different tracing strategies
• Examples of using Fay for performance analysis
• Fay is based on LINQ and Windows specifics
• Could build on Linux using Ftrace, Hadoop, etc.
• Some restrictions apply currently
• E.g., skew towards batch processing due to Dryad

• Fay: Flexible tracing of distributed executions
• Both expressiveand efficient
• Unified trace queries
• Pervasive data-parallelism
• Safe machine-code probe processing
• Often equally efficient as purpose-built tools

from ioin cluster.Function("iolib!Read")

where io.time < Now.AddMinutes(5)

let size = io.Arg(2) // request size in bytes

group ioby size/1024 into g

select new { sizeInKilobytes = g.Key,

countOfReadIOs = g.Count() };

• Aggregates read activity in iolib module
• Across cluster, both user-mode & kernel
• Over 5 minutes

from ioin cluster.Function("iolib!Read")

where io.time < Now.AddMinutes(5)

let size = io.Arg(2) // request size in bytes

group ioby size/1024 into g

select new { sizeInKilobytes = g.Key,

countOfReadIOs = g.Count() };

• Specifies what to trace
• 2nd argument of read function in iolib
• And how to aggregate
• Group into kb-size buckets and count