Dagstuhl Seminar on Event Processing

1. Dagstuhl Seminar on Event Processing Deep Dive - WG 3 Event Processing and the Rest of the IT World

2. Charter The WG works on the relationships of event processing with other areas like: • Databases • Rules • BPM • Analytics • Cloud computing • Social computing • …

3. Our trajectory • Initial charter • Event processing reach • Top-k interests • Questions we asked • Characterizing event processing • Ancestors, contemporaries, consumers • Ancestors • Contemporaries • Consumers

4. Event Processing Reach View Maintenance Business Process Management Semantic Web Trading Systems Databases Messaging Modeling Formal Modeling Log processing Low Latency Middleware Interactive Systems Distributed Caching EDA Distributed Sensors Analytics HCI ECA rules H.P.C. Main Memory DB's RFID Middleware Context Modeling Governance Publish Subscribe Systems Spatial Databases Math modeling Context processing on historical data ESB Business Rules Data Mining SOA Programming Languages Integration Architecture Text Mining Temporal Databases Situation Awareness Computer Security The size of the tags in the cloud have no meaning and have been picked artisitically

5. Top-k interests • Analytics 15 votes • Business Rules 15 votes • Publish Subscribe 15 votes • Situation Awareness 13 votes • Business Process Management 13 votes • Business Activity Monitoring 11 votes • ECA rules 11 votes • Messaging 11 votes • … Voting figures taken on Tuesday 2010/05/18 09:25

6. Questions we initially aimed for • Is there anything new in event processing (EP)? • Can EP solve existing problems in a better way? • How is EP distinguished from predecessors and contemporaries? • What problems require EP & what of EP do they require? • What are the underlying technologies & concepts that EP builds upon?

7. Anything New in EP? • “The whole is more than the sum of its parts.” • The events treated as first class citizen, not added a posteriori. • Temporal aspect integral part of event processing • EP’s core functionalities are filtering, matching, detecting, aggregating, & abstracting • Feels the natural solution for the problem

8. EP (platform) tends to • Feel more likean inverse database • Event expression base • Maybe a filterbase • Processing of events against filters vs. • Queries against data tuples

9. Inverse database query events data tuples expressions About past About future events sets of tuples Query and expressions are very similar. Data tuples and events are very similar. However, the two problem statements are inverse.

10. EP tends to • Implement • Event expression index • Reuse of (partial) processing state • Amortize cost across shared processing • Share output and partial results

11. EP tends to • Follow a continuous operational model • Processing of live events • Processing in real-time • Processing of logged events • Processing of a priori infinite flows of events • Create meaningful aggregations & abstractions • Event(s) in, event(s) out (event algebra) • Look and feel like push-based processing (implementation is a different story) • Be reactive (i.e., event-driven, must react to event now as opposed to whenever)

12. EP Ancestors • ECA & active databases (DB)  • Incremental view maintenance • Discrete event simulation • Event correlation (network mgmt.) • Expert systems (AI) • Temporal databases (DB) • HCI, i.e., GUI event loops (SE) • Theories of events (philosophy)

13. Event Condition Action (ECA) • ECA is one of the origins of event processing • ECA goes with active databases • Event(s), conditions, actions • Separates events from production rules • Makes events explicit, as opposed to facts (knowledge) in inference & production systems Klaus R. Dittrich, Stella Gatziu, Andreas Geppert: The Active Database Management System Manifesto: A Rulebase of ADBMS Features. Lecture Notes in Computer Science 985, Springer 1995, ISBN 3-540-60365-4, pages 3-20.

14. Event Processing ECA ECA & EP • ECA • Is embraced by event processing • ECA rules do not cover event stream processing, complex events Event stream processing Complex Event 

15. Contemporaries • Publish/Subscribe  • Stream processing  • Rule-based processing • Continuous query processing • Incremental view maintenance

16. Publish/Subscribe 101 • Not all publish/subscribe is equal! • Publish/Subscribe models and evolution • Channel-based • OMG CORBA Event Service, … • Topic-based • WS Notifications, OMG Data Dissemination Service … • Type-based • OMG Data Dissemination Service (partially), … • Content-based • The PADRES ESB (see below), … • State-based • Subject Spaces CANOE Summer School, Norway, 2009

17. Publish/Subscribe • Encompasses event modeling, event filtering, event detection and event dissemination • Pub/Sub stands for decoupling • Pub/Sub enables loose coupling

18. Pub/Sub & EP • Decoupling (required in pub/sub) is an orthogonal concern in EP • Event dissemination (integral part of pub/sub) is an orthogonal concern in EP • EP & pub/sub share filtering, matching, detection and aggregation functionalities • EP & pub/sub share requirements for specifying events

19. Pub/Sub & EP • Publish/Subscribe • Decoupling (required in pub/sub) is an orthogonal concern in EP • Event dissemination (integral part of pub/sub) is an orthogonal concern in EP • EP & pub/sub share filtering, matching, detection and aggregation functionalities • EP & pub/sub share requirements for specifying events Event Processing Publish Subscribe

20. Streams • Streams are infinite, time-varying relations over temporally ordered tuples • Strict tuple order (not a requirement in every product) • Tuple assumed to at least have an implicit timestamp • In good DB manner, streams employ a fixed, rigid, homogenous schema • Stream DBs generally process a handful of queries at the same time • But, what about video, audio, ... stream processing? • Data flowing through a data flow graph

21. Streams Streams Event Processing • Streams are an instance of event processing • Converging the same areas • Some differences relating to the treatment of time • Physical reality & problems with causality • Distributed sources • Non-synchronized clocks • Source-based timestamp

22. Streams • Streams • Beside time/clock-synchronization issues, streams can be seen as a direct instantiation of event processing • Often referred to as Event Stream Processing Event Processing Streams

23. Consumers • Business Process Management • Analytics View Maintenance Business Process Management Semantic Web Trading Systems Databases Messaging Modeling Formal Modeling Log processing Low Latency Middleware Interactive Systems Distributed Caching EDA Distributed Sensors Analytics HCI ECA rules H.P.C. Main Memory DB's RFID Middleware Context Modeling Governance Publish Subscribe Systems Spatial Databases Math modeling Context processing on historical data ESB Business Rules Data Mining SOA Programming Languages Integration Architecture Text Mining Temporal Databases Situation Awareness Computer Security

24. Business Process Management • … which we took to mean the (modeling), execution, monitoring of business processes Store inDB … Loan Application Processing Reject gid=c001 Creditcheck 2 gid=c001 gid=c001 end Checkscore Checkscore 2 Creditcheck Approve Send toofficer else … else

25. BPM’s use of EP • EP solutions exist for BP orchestration & choreography • EP adds agility, flexibility and adaptivity to BPM by opening up the BP black-box (enables ad hoc process changes) • EP enables Business Activity Monitoring (BAM): real-time process monitoring

26. BPM & EP Process Execution EP enables ad hoc business processes & ad hoc changes to processes Business Process Management Event Processing Monitoring Continuous process monitoring.

27. Analytics • Application of predictive techniques, data mining, learning, and customer scoring models • Operational (real-time) vs. strategic (mining) • Examples • Fraud detection • Detect specified patterns • Discover new patterns • Refine patterns from new input • Shopping-, scoring-, behavior-model, • Medical-, network-, security-monitoring • Financial risk and compliance analysis

28. Analytics Analytics Event Processing • Operational: • Event detection executed in real-time while • Analytics uses EP to execute queries for operational analysis • EP operates over continuous (live) data • EP creates meaningful aggregation • EP is a natural model for continuous event detection • EP model inverse database model • EP abstracts from lower-level events • Temporal aspects is evidence for use of EP • EP’s core functionalities are filtering, matching and aggregation • Strategic: • Information gathering is based on historical data • EP uses analytics for: • Specification of existing patterns • Learning of new patterns • Refining existing and new patterns • There is a line between the statistical learning and EP

29. Analytics & EP Operational Overlap: Use of EP to support monitoring during the occurrence (e.g. via continuous processing) Analytics Event Processing • Strategic Overlap: • Usage of historical data to: • Specify existing, • Detect new and • Refine existing patterns

30. “Thief” detection Video feeds Pre-processing (feature extraction) EP describing typical movement patterns Alert Dispatch security EP application modeling Pro EP • Create added value by enabling • integration of other sensors • EP abstractions • help build this • application? • Lower-level use of EP too • cumbersome • Right level of use • of EP unclear Detect arm movement Con EP Sequence of (arm, X, Y) coordinates Image processing generated

31. In retrospect • Diverse group, broad point of view, yet we agreed on many core EP characteristics • Event processing and event-based systems • In the end it is a question of degrees • Based on a catalogue of characteristics (evidence)