Measurement Flow Architecture in OML

Jolyon White GEC9, 4th November 2010 Measurement Flow Architecture in OML

OML = Measurement Flows Rutgers University, New Jersey Parking Discovery Rutgers Marco Gruteser Deutsche Telekom Labs @ TU Berlin BOWL Testbed National Broadband Network 100Mbs FTTH VoD Trial Rail Bridge Monitoring Sensors NSW Road Traffic Authority IREEL Network Education Teaching Platform NICTA, Sydney

Current OML data pipeline Measurement points Filters Measurement streams Database tables OML Server Database (SQL) Application or Service File OML client library

Schemas • Schemas enable: • Provenance • Processing in the pipeline (data crunching) • Measurement Stream schema == Combination of schemas of filter outputs • Each MS stored in its own DB table A (S, T) MS Schema MP (A, B, C) B (U, V, W) (S, T, U, V, W, X, Y) C (X, Y)

Schemas • Example: app name is “otr2” • SQL issued to the database: • Schema names + metadata define provenance MP udp_in: avg : DOUBLE max : DOUBLE min : DOUBLE avg ts : DOUBLE flow_id : INT32 seq_no : UINT32 pkt_length : UINT32 src_host : STRING src_port : STRING CREATE TABLE otr2_udp_in ([METADATA COLS],pkt_length_avgREAL, pkt_length_max REAL, pkt_length_min REAL);

Measurement Collection Graph • Modularize producers + consumers • Measurement Point (MP) – data source • Processing Point (PP) – buffer, select, filter, join, forward • Termination Point (TP) – persistent storage PP PP PP MP MP MP TP TP TP Metadata Store Services API MDA (Measurement Data Archive)

Resource provisioning • OML has no concept of resource provisioning • Experimenter obtains resources for I&M identically to experimental resources • i.e. no distinction between I&M and experiment resources • User has full control over how resources used • Useful defaults, but allow more if experimenter wants it • Can’t always cleanly separate I&M from experiment • Mobile wireless testbeds where I&M must share compute + network with experiment • E.g. Parknet • Almost all wireless traffic was measurement flows

Transports • OML currently supports two custom procotols • Text version • Binary version • Standard transports are good! • We like IPFIX, aiming to support it (near future) • Why? Several reasons but: • Template support  self-describing measurement streams Metadata headers (schemas) Measurement flow Metadata headers (schemas) Measurement flow

Processing Point Applications

Proxy Server • Buffer measurements on command • Don’t transmit to remote server • Same protocol as server • Transparent to client applications CMD_BUFFER CMD_REPLAY Application OML Server Proxy server

Hierarchical Measurement Collection • High-resolution measurements lose value over time • Local storage may be limited • Measuring at different granularities • Inspired by existing research in Streaming Databases • Numerous VC-backed startups in financial data feed processing space

Context-Driven Experiment Steering • Dynamic experiments need measured context feedback • E.g. Geographic trip lines, link state feedback

Context-Driven Measurement • Environment feedback can be used to influence the measurement process itself

Measurement Flow Architecture in OML