Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010 - PowerPoint PPT Presentation

benjamin
event processing past present future vldb 2010 tutorial singapore september 15 th 2010 n.
Skip this Video
Loading SlideShow in 5 Seconds..
Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010 PowerPoint Presentation
Download Presentation
Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010

play fullscreen
1 / 90
Download Presentation
Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010
484 Views
Download Presentation

Event processing – past, present, future VLDB 2010 Tutorial, Singapore, September 15 th , 2010

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Event processing – past, present, futureVLDB 2010 Tutorial, Singapore, September 15th, 2010 Opher Etzion (opher@il.ibm.com)

  2. An example to kick-off this tutorial: Event processing can help here.. THE LUGGAGE PERSPECTIVE: Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found. BBC News, April 4, 2007 Act: Passenger has been re-routed to another destination – send the luggage Notify: Bag has been checked but did not reach the ULD within 20 minutes Notify: Bag has been checked but did not reach the connecting flight Act: Bag has reached to the wrong aircraft

  3. Another example: premature baby monitoring Patient is hooked up to multiple monitors (in hospital or at home) - the physician can set up event- based rules on multiple measurements and patient’s history when to send an alert and to whom: Defined Pattern

  4. Outline of this tutorial What is behind event processing ? How is it related to other computing terms? Where are its roots? Event processing – architecture, building blocks The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications The future: Trends and research challenges

  5. What is behind event processing ? How is it related to other computing terms? Where are its roots?

  6. What is “event processing” anyway? or Event processing is a form of computing that performs operations on events

  7. In computing we ptocessed events since early days Network and System Management

  8. Emerging technologies in enterprise computing(Gartner Hype Cycle, Summer 2009)

  9. What’s new? The analog: moving from files to DBMS In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that was traditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.

  10. What is an event There are various definitions of events Event is an occurrence within a particular system or domain; the word event has double meaning: the real-world Occurrence as well as its computerized representation

  11. In daily life we often react to events..

  12. Sometime we even react to the occurrence of multiple events I closed the deal with the Australians I closed the deal with the Canadians We closed two huge deals in a single day, It is a good opportunity to send all the team to Some fun time in Singapore

  13. Pattern detection is one of the notable functions of event processing Event Patterns

  14. What we actually want to react to are – situation Toll violation Frustrated customer TOLL VILOATOR Sometimes the events can approximate or indicate with some certainty that the situation has occurred Sometimes the situation is determined by detecting that some pattern occurred in the Flowing events. FRUSTRATED CUSTOMER

  15. Event processing is being used for various reasons

  16. Event Driven Architecture Event driven architecture: asynchronous, decoupled; each component is autonomic.

  17. Ancestor: active databases Composite events were inherited to event processing On event When condition Do action With coupling mode

  18. Ancestor: Data Stream management system Source: Ankur Jain’s website

  19. Ancestor: Temporal databases There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added

  20. Ancestor: Messaging – pub/sub middleware

  21. Ancestor: Discrete event simulation

  22. Ancestor: Network and system management

  23. Event processing – architecture, building blocks

  24. Location Service Bid System Fast Flower Delivery Delivery Request Location GPS Location Bid Request Store Preferences Ranked drivers / automatic assignment Delivery Bid Manual Assignment Assignment Assignment System Flower Store Assignments, Bid alerts, Assign Alerts Van Driver Pick Up confirmation Delivery confirmation Ranking and Reporting System Assignments, Pick Up Alert Delivery Alert Driver’s Guild Control System Ranking and reports

  25. Event processing network

  26. Example of EPN – part of the FFD example

  27. The seven Building blocks Event Channel Event Type Event Consumer Event Producer Event Processing Agent Context Global State

  28. Event type definition

  29. Producer – State Observer in workflows State observer Push: Instrumentation points; Pull: Query the state

  30. Producer – Code instrumentation

  31. Producer – syndication

  32. Producers – streams to events

  33. Producer – sensors

  34. Consumer - Performance monitoring dashboard

  35. Consumer - Ambient Orb

  36. Producer and consumer - Sixth sense

  37. Twitter as a consumer

  38. Event Processing Agents Event Processing Agent Filter Transform Detect Pattern Translate Aggregate Split Compose Enrich Project

  39. The EPA picture

  40. A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event. Filter EPA

  41. Transform EPA sub types

  42. Sample of pattern types • all pattern is satisfied when the relevant event set contains at least one instance of each event type in the participant set • any pattern is satisfied if the relevant event set contains an instance of any of the event types in the participant set • absence pattern is satisfied when there are no relevant events • relative N highest values pattern is satisfied by the events which have the N highest value of a specific attribute over all the relevant events, where N is an argument • value average pattern is satisfied when the value of a specific attribute, averaged over all the relevant events, satisfies the value average threshold assertion. • always pattern is satisfied when all the relevant events satisfy the always pattern assertion • sequence pattern is satisfied when the relevant event set contains at least one event instance for each event type in the participant set, and the order of the event instances is identical to the order of the event types in the participant set. • increasing pattern is satisfied by an attribute A if for all the relevant events, e1 << e2 e1.A < e2.A • relative max distance pattern is satisfied when the maximal distance between any two relevant events satisfies the max threshold assertion • moving toward pattern is satisfied when for any pair of relevant events e1, e2 we have e1 << e2  the location of e2 is closer to a certain object then the location of e1.

  43. Find the five highest bids within the bid interval Pattern detection example Pattern name: Manual Assignment Preparation Pattern Type: relative N highest Context: Bid Interval Relevant event types: Delivery Bid Pattern parameter: N = 5; value = Ranking Cardinality: Single deferred Taken from the Fast Flower Delivery use case

  44. Why do we need policies - A simple example: heavy trading scenario Why defining patterns is not that easy? Because we need to tune up the semantics • Given: • A stream of events of a single topic, about the activity in the stock market for a certain stock. • An event is produced every 10 minutes when there is trade in the stock. • Each event consists of: quote (current stock-quote), volume (an accumulated volume of traded events within these 10 minutes). • A selection specification: “trigger an automatic trade program if the volume exceeds 300,000 3 times within an hour; pass as an argument the last quote and the sum of the 3 volume values”. • How many times the trade programming is triggered ; • Which arguments are used in each triggering?

  45. Pattern policies Evaluation policy—This determines when the matching sets are produced Deferred Cardinality policy—This determines how many matching sets are produced within a single context partition Repeated type policy—This determines what happens if the matching step encounters multiple events of the same type Every - Override – First – Last – with max/min value of…

  46. Pattern policies – cont. Consumption policy—This specifies what happens to a participant event after it has been included in a matching set Consume Bounded reuse Reuse Order policy—This specifies how temporal order is defined By occurrence time – by detection time – by stream position – by attribute

  47. Our entire culture is context sensitive • In the play “The Tea house of the August Moon” one of the characters says: Pornography question of geography • This says that in different geographical contexts people view things differently • Furthermore, the syntax of the language (no verbs) is typical to the way that the people of Okinawa are talking When hearing concert people are not talking, eating, and keep their mobile phone on “silent”.

  48. Context has three distinct roles (which may be combined) The events that relate to each customer are processed separately Partition the incoming events Grouping together events that happened in the same hour at the same location Grouping events together Different processing for Different context partitions Determining the processing

  49. Temporal Spatial State Oriented Segmentation Oriented Context Definition Acontext is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions. A context may have one or more context dimensions.

  50. Context Types Examples Segmentation Oriented “All Children 2-5 years old” “All platinum customers” Temporal Spatial “Every day between 08:00 and 10:00 AM” “A week after borrowing a disk” “A time window bounded by TradingDayStart and TradingDayEnd events” “3 miles from the traffic accident location” “Within an authorized zone in a manufactory” Context State Oriented “Airport security level is red” “Weather is stormy”