StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time

Download Presentation

StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time

Loading in 2 Seconds...

- 177 Views
- Uploaded on
- Presentation posted in: General

StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

StatStream: Statistical Monitoring of Thousands of Data Streams in Real Time

Pankaj Kumar Madhukar

Rakesh Kumar Singh

Puspendra Kumar

Project Instructor:

Prof P.K.Reddy

Correlated!

Correlated!

- Given tens of thousands of high speed time series data streams, to detect high-value correlation, including synchronized and time-lagged, over sliding windows in real time.

- Real time
- high update frequency of the data stream
- fixed response time, online

- Naive algorithm
- N : number of streams
- w : size of sliding window
- space O(N) and time O(N2w) VS space O(N2) and time O(N2) .

- Suppose that the streams are updated every second.
- With a Pentium 4 PC, the exact computing method can only monitor 700 streams with a delay of 2 minutes.

- Our Approach
- Using Discrete Fourier Transform to approximate correlation
- Using grid structure to filter out unlikely pairs
- Our approach can monitor 10,000 streams with a delay of 2 minutes.

- Goal
- StatStream
- Data Structure
- Correlation Approximation
- Grid structure

- Empirical study
- Future work

Basic window digests:

sum

DFT coefs

Basic window digests:

sum

DFT coefs

Time point

Basic window

Sliding window

- Three level time interval hierarchy
- Time point, Basic window, Sliding window

- Basic window (the key to our technique)
- The computation for basic window i must finish by the end of the basic window i+1
- The basic window time is the system response time.

- Digests

Basic window digests:

sum

DFT coefs

Basic window digests:

sum

DFT coefs

Sliding window digests:

sum

DFT coefs

- Motivation and Goal
- Related work
- StatStream
- Data Structure
- Correlation Approximation
- Grid structure

- Empirical study
- Future work

- Inner-product of aligned basic windows

Stream x

Stream y

Basic window

Sliding window

f1(1) f1(2) f1(3) f1(4) f1(5) f1(6) f1(7) f1(8)

y1 y2 y3 y4 y5 y6 y7 y8

f2(1) f2(2) f2(3) f2(4) f2(5) f2(6) f2(7) f2(8)

f3(1) f3(2) f3(3) f3(4) f3(5) f3(6) f3(7) f3(8)

- Approximate with an orthogonal function family (e.g. DFT)
- Inner product of the time series Inner product of the digests
- The time and space complexity is reduced from O(b) to O(n).
- b : size of basic window
- n : size of the digests (n<<b)

- e.g. 120 time points reduce to 4 digests

x1 x2 x3 x4 x5 x6 x7 x8

sliding window

sliding window

- Inner-product with unaligned windows

- The time complexity is reduced from O(b) to O(n2) , as opposed to O(n) for synchronized correlation.

- Motivation and Goal
- Related work
- StatStream
- Data Structure
- Correlation Approximation
- Grid structure

- Empirical study
- Future work

x

- The DFT coefficients yields a vector.
- High correlation => closeness in the vector space
- We can use a grid structure and look in the neighborhood, this will return a super set of highly correlated pairs.

- Motivation and Goal
- Related work
- StatStream
- Data Structure
- Correlation Approximation
- Grid structure

- Empirical study
- Future work

- Response time
- Exact (naïve method): T=k0bN2

- DFT-grid:
- Updating Digests: T1=k1bN
- Detecting correlation:T2=k2N2

- Approximation errors
- Larger size of digests, larger size of sliding window and smaller size of basic window give better approximation
- The approximation errors are small for the stock data.

- Precision: the quality of the grid structure

- Motivation and Goal
- Related work
- StatStream
- Data Structure
- Correlation Approximation
- Grid structure

- Empirical study
- Future work

- Algorithmic:
- dynamic clustering of streams
- outlier detection
- a stream that becomes less correlated with the other streams in its cluster.

- Applications:
- Data-intensive application requiring correlation among many streams.
- Network Traffic Monitoring:
- The unusual high correlation between two links in a network might suggest some anomaly.

- Medical Time Series:
- The high correlation between the two region in the human brain during fMRI testing might suggest some functional connection.

- Some domain specific definition of correlation might be more appropriate.
- E.g., in fMRI time series, detrending before correlating.