Download
Download Presentation
Loading in 2 Seconds...
Fay. Extensible Distributed Tracing from Kernels to Clusters. Úlfar Erlingsson, Google Inc. Marcus Peinado , Microsoft Research Simon Peter, Systems Group, ETH Zurich Mihai Budiu , Microsoft Research. Wouldn’t it be nice if…. We could know what our clusters were doing?
Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.
While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.
Úlfar Erlingsson, Google Inc.
Marcus Peinado, Microsoft Research
Simon Peter, Systems Group, ETH Zurich
MihaiBudiu, Microsoft Research
var kernelFunctionFrequencyVectors =
cluster.Function(kernel, “syscalls!*”)
.Where(evt => evt.time < Now.AddMinutes(3))
.Select(evt => new { Machine = fay.MachineID(),
Interval = evt.Cycles / CPS,
Function = evt.CallerAddr })
.GroupBy(evt => evt,
(k,g) => new { key = k, count = g.Count() });
Vector Nearest(Vector pt, Vectors centers) {
var near = centers.First();
foreach (var c in centers)
if (Norm(pt – c) < Norm(pt – near))
near = c;
return near;
}
Vectors OneKMeansStep(Vectors vs, Vectors cs) {
return vs.GroupBy(v => Nearest(v, cs))
.Select(g => g.Aggregate((x,y)
=> x+y)/g.Count());
}
Vectors KMeans(Vectors vs, Vectors cs, int K) {
for (int i=0; i < K; ++i)
cs = OneKMeansStep(vs, cs);
return cs;
}
var kernelFunctionFrequencyVectors =
cluster.Function(kernel, “syscalls!*”)
.Where(evt => evt.time < Now.AddMinutes(3))
.Select(evt => new { Machine = fay.MachineID(),
Interval = evt.Cycles / CPS,
Function = evt.CallerAddr })
.GroupBy(evt => evt,
(k,g) => new { key = k, count = g.Count() });
Fay: Flexible monitoring of distributed executions
var kernelFunctionFrequencyVectors =
cluster.Function(kernel, “*”)
.Where(evt => evt.time < Now.AddMinutes(3))
.Select(evt => new { Machine = fay.MachineID(),
Interval = evt.Cycles / CPS,
Function = evt.CallerAddr})
.GroupBy(evt => evt,
(k,g) => new { key = k, count = g.Count() });
var kernelFunctionFrequencyVectors =
cluster.Function(kernel, “*”)
.Where(evt => evt.time < Now.AddMinutes(3))
.Select(evt => new { Machine = MachineID(),
Interval = w.Cycles / CPS,
Function = w.CallerAddr})
.GroupBy(evt => evt,
(k,g) => new { key = k, count = g.Count() });
Vector Nearest(Vector pt, Vectors centers) {
var near = centers.First();
foreach (var c in centers)
if (|pt – c| < |pt – near|)
near = c;
return near;
}
Vector Nearest(Vector pt, Vectors centers) {
var near = centers.First();
foreach (var c in centers)
if (Norm(pt – c) < Norm(pt – near))
near = c;
return near;
}
Vectors OneKMeansStep(Vectors vs, Vectors cs) {
return vs.GroupBy(v => Nearest(v, cs))
.Select(g => g.Aggregate((x,y)
=> x+y)/g.Count());
}
Vectors KMeans(Vectors vs, Vectors cs, int K) {
for (int i=0; i < K; ++i)
cs = OneKMeansStep(vs, cs);
return cs;
}
Vectors OneKMeansStep(Vectors vs, Vectors cs) {
return vs.GroupBy(v => Nearest(v, cs))
.Select(g => g.Aggregate((x,y)
=> x+y)/g.Count());
}
Vectors KMeans(Vectors vs, Vectors cs, int K) {
for (int i=0; i < K; ++i)
cs = OneKMeansStep(vs, cs);
return cs;
}
K-Means: System calls
K-Means: Clustering
K-Means: System calls
K-Means: Clustering
Fay: Flexible monitoring of distributed executions
Fay: Flexible monitoring of distributed executions
Tracing Runtime
query
Createprobe
ETW
User-Space
Kernel
Target
Fay
Probe
XFI
Hotpatching
200 cycles
Module with a traced function Foo
Caller:
...
e8ab62ffff call Foo
...
ff1508e70600 call[Dispatcher]
Foo: ebf8 jmp Foo-6
cccccc
Foo2: 57 push rdi
...
c3 ret
Module with a traced function Foo
Fay platform module
Dispatcher:
t = lookup(return_addr)
...
call t.entry_probes
...
call t.Foo2_trampoline
...
call t.return_probes
...
return /* to after call Foo */
Caller:
...
e8ab62ffff call Foo
...
ff1508e70600 call[Dispatcher]
Foo: ebf8 jmp Foo-6
cccccc
Foo2: 57 push rdi
...
c3 ret
Module with a traced function Foo
Fay platform module
Fay probes
Dispatcher:
t = lookup(return_addr)
...
call t.entry_probes
...
call t.Foo2_trampoline
...
call t.return_probes
...
return /* to after call Foo */
Caller:
...
e8ab62ffff call Foo
...
ff1508e70600 call[Dispatcher]
Foo: ebf8 jmp Foo-6
cccccc
Foo2: 57 push rdi
...
c3 ret
PF3
XFI
PF4
PF5
XFI
XFI
Slowdown (x)
Null-probe overhead
Cycles
from ioin cluster.Function("iolib!Read")
where io.time < Now.AddMinutes(5)
let size = io.Arg(2) // request size in bytes
group ioby size/1024 into g
select new { sizeInKilobytes = g.Key,
countOfReadIOs = g.Count() };
from ioin cluster.Function("iolib!Read")
where io.time < Now.AddMinutes(5)
let size = io.Arg(2) // request size in bytes
group ioby size/1024 into g
select new { sizeInKilobytes = g.Key,
countOfReadIOs = g.Count() };
