Recent Results in Resource Signal Measurement, Dissemination, and Prediction

1. Sensor Data Data Ethernet IP TCP Padding • Overwrite unused or redundant fields with sensor data • Shannon entropy of packet headers: 4.8 bits per byte • In practice: 17..32 bits at IP and TCP, padding varies • Implementations: Minet, Linux Kernel Modules • [LCR 2002, NWU-CS-02-12] • http://www.cs.northwestern.edu/~plab/Diffusion Diffusion: Zero Cost Information Dissemination (with Brian Cornell, Jack Lange, NSF REU) Sensor App App Consumer Transport Transport Network Network Header Editing Data Extraction Data Link Data Link Physical Physical • Sensor data piggybacked on existing application packets • Number and size of packets unchanged Recent Results in Resource SignalMeasurement, Dissemination, and Prediction Peter A. DindaPrescienceLabDepartment of Computer ScienceNorthwestern University .cs.northwestern.edu RPS: The Resource Prediction System Toolkit • RPS: Measure, Predict, and Disseminate information about dynamic resource supply • Ultimate goal: provide advice to adaptive applications • Publicly available, Extensible, Portable, Easy buy-in • Resource signals: Discrete-time signals strongly correlated with resource supply • Host load • Windows performance counters (using WatchTower) • Network flow bandwidth and latency (using Remos) • Any text-based source • Online predictive modeling • Simple models (MEAN, BESTMEAN, BESTMEDIAN, LAST…) • Box/Jenkins Models (AR, MA, ARMA, ARIMA,…) • Fractional ARIMAs • Nonlinear modeling (TARs, Wavelet-decompositions) • [HPDC 99, Cluster 00, Cluster 02, SIGMETRICS 01, IPDPS 02, SC 01, SHAMAN 02] • http://www.cs.northwestern.edu/~RPS Tsunami: Wavelet-based Approaches (with Jason Skicewicz) Video App Sensor Fine-grain measurement … Resource-appropriate measurement Grid App Resource Signal (periodic sampling) Example: host load Course-grain measurement • Tension between different application needs • Application and sensor needs coupled • Inefficient bandwidth usage, especially in unicast Application Sensor Multicast Multiscale Prediction of Network Bandwidth (with Yi Qiao, Jason Skicewicz) Stream Interval Level 0 Level 0 • Large study of predictability of binned packet traces • Offline RPS predictors (linear models) • Different resolutions • Both power-of-two binning and low-pass via D8 wavelets • Over 200 NLANR and other traces • Mostly WANs • All long period traces available at time of trace • Random selection of short-term traces • Hierarchical classification of traces • Short period, Long Period, Bellcore • Predictability using linear models highly variable • Many traces unpredictabile white noise • Predictability varies with resolutiion • Sweet Spot: Predictability often maximized at particular resolution • [NWU-CS-02-12, NWU-CS-02-13] • http://www.cs.northwestern.edu/~plab/Clairvoyance Wavelet Transform Inverse Wavelet Transform Level L Level M-1 Level M • Sensor sends all levels appropriate to sampling rate • Each application receives levels based on its needs • Applications and sensors decoupled • Level rates decrease logarithmically. • Limited proof of concept implemented in RPS: Works • Wavelet toolbox in next RPS release • Biggest issue: The Delay Problem • Transforms introduce sample delay • Exponential in the number of levels • Affects both streaming and block transforms • Current efforts to overcome the delay problem • Exploit prediction (limited success so far) • Exploit “wavelet-like” decompositions that can trade-off between reconstruction accuracy and delay • [HPDC 2001] • http://www.cs.northwestern.edu/~plab/Tsunami