application of event stream processing toward qa qc in gleon n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Application of Event Stream Processing toward QA/QC in GLEON PowerPoint Presentation
Download Presentation
Application of Event Stream Processing toward QA/QC in GLEON

Loading in 2 Seconds...

play fullscreen
1 / 11
weylin

Application of Event Stream Processing toward QA/QC in GLEON - PowerPoint PPT Presentation

110 Views
Download Presentation
Application of Event Stream Processing toward QA/QC in GLEON
An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Application of Event Stream Processing toward QA/QC in GLEON Sameer Tilak, Chris Solomon, Peter Shin, Tony Fountain, and Peter Arzberger.

  2. Science: Ability to process and manage large amounts of data in a scalable and efficient fashion • Engineering/Operations and Management: Reliable and smooth operation of the system Motivation

  3. Non Real-Time (Traditional) QA/QC Approach • Set up an experiment, collect data in flat files/ database. • Look at acquired data after few days/weeks/months. • QA/QC is done either: • Manually: Visual inspection • Automated: Matlab/Perl scripts etc.

  4. Sample QA/QC Procedure • Out-of-range values • Missing data • Sensor flat lines • Signal shifts etc.

  5. Issues with current approach • Real World is Messy. QA/QC might take a lot of time! Even up to 50% • System Level: Clock drifts, batteries fail, sensors malfunction, servers and networks fail. You can’t respond to these in real-time • Application/Science Level: Rear event occurs, abnormal event happens you can’t respond to it in real-time

  6. Event Stream Processing (ESP) Overview • ESP and CEP have emerged out of years of research in the field of active database, continuous queries and stream processing and are now being widely used in modern latency-sensitive architectures such as algorithmic trading systems, fraud or intrusion detection and real-time business intelligence or customer relationship management.

  7. Real-Time Event Stream Processing (ESP) • Database up-side down • Persistent Queries versus single-shot queries • Seamless integration of persistent data with streaming data • Time-windows are user-defined can span anywhere from few milliseconds to hours or days

  8. Proposed Architecture

  9. Advantages of the Proposed Architecture • Real-Time notification -- on the order of few seconds/minutes • Scalable and modular architecture • Enables new types of science • Novel system monitoring capabilities • Essential QA/QC happens in more efficient manner e.g. pre-processing.

  10. Tsunami Data

  11. Future Work • Work more with domain scientists to fine-tune QA/QC procedures. • Employ Cloud Computing. • Live-Migration of virtual machines in real-time in response to critical events to provide more compute power, network bandwidth etc.