Event processing – past, present, futureVLDB 2010 Tutorial, Singapore, September 15th, 2010 Opher Etzion (firstname.lastname@example.org)
An example to kick-off this tutorial: Event processing can help here.. THE LUGGAGE PERSPECTIVE: Across the 24 largest airlines more than 5.6 million bags went missing in 2006, this is an average of 15.7 bags per 1,000 travelers. 15% of the bags are never found. BBC News, April 4, 2007 Act: Passenger has been re-routed to another destination – send the luggage Notify: Bag has been checked but did not reach the ULD within 20 minutes Notify: Bag has been checked but did not reach the connecting flight Act: Bag has reached to the wrong aircraft
Another example: premature baby monitoring Patient is hooked up to multiple monitors (in hospital or at home) - the physician can set up event- based rules on multiple measurements and patient’s history when to send an alert and to whom: Defined Pattern
Outline of this tutorial What is behind event processing ? How is it related to other computing terms? Where are its roots? Event processing – architecture, building blocks The present: State of the practice in event processing – languages, implementation issues, challenges in implementing event processing applications The future: Trends and research challenges
What is behind event processing ? How is it related to other computing terms? Where are its roots?
What is “event processing” anyway? or Event processing is a form of computing that performs operations on events
In computing we ptocessed events since early days Network and System Management
Emerging technologies in enterprise computing(Gartner Hype Cycle, Summer 2009)
What’s new? The analog: moving from files to DBMS In recent years – architectures, abstractions, and dedicated commercial products emerge to support functionality that was traditionally carried out within regular programming. For some applications it is an improvement in TCO; for others is breaking the cost-effectiveness barrier.
What is an event There are various definitions of events Event is an occurrence within a particular system or domain; the word event has double meaning: the real-world Occurrence as well as its computerized representation
Sometime we even react to the occurrence of multiple events I closed the deal with the Australians I closed the deal with the Canadians We closed two huge deals in a single day, It is a good opportunity to send all the team to Some fun time in Singapore
Pattern detection is one of the notable functions of event processing Event Patterns
What we actually want to react to are – situation Toll violation Frustrated customer TOLL VILOATOR Sometimes the events can approximate or indicate with some certainty that the situation has occurred Sometimes the situation is determined by detecting that some pattern occurred in the Flowing events. FRUSTRATED CUSTOMER
Event Driven Architecture Event driven architecture: asynchronous, decoupled; each component is autonomic.
Ancestor: active databases Composite events were inherited to event processing On event When condition Do action With coupling mode
Ancestor: Data Stream management system Source: Ankur Jain’s website
Ancestor: Temporal databases There is a substantial temporal nature to event processing. Recently – also spatial and spatio-temporal functions are being added
Location Service Bid System Fast Flower Delivery Delivery Request Location GPS Location Bid Request Store Preferences Ranked drivers / automatic assignment Delivery Bid Manual Assignment Assignment Assignment System Flower Store Assignments, Bid alerts, Assign Alerts Van Driver Pick Up confirmation Delivery confirmation Ranking and Reporting System Assignments, Pick Up Alert Delivery Alert Driver’s Guild Control System Ranking and reports
The seven Building blocks Event Channel Event Type Event Consumer Event Producer Event Processing Agent Context Global State
Producer – State Observer in workflows State observer Push: Instrumentation points; Pull: Query the state
Event Processing Agents Event Processing Agent Filter Transform Detect Pattern Translate Aggregate Split Compose Enrich Project
A filter EPA is an EPA that performs filtering only, and has no matching or derivation steps, so it does not transform the input event. Filter EPA
Sample of pattern types • all pattern is satisfied when the relevant event set contains at least one instance of each event type in the participant set • any pattern is satisfied if the relevant event set contains an instance of any of the event types in the participant set • absence pattern is satisfied when there are no relevant events • relative N highest values pattern is satisfied by the events which have the N highest value of a specific attribute over all the relevant events, where N is an argument • value average pattern is satisfied when the value of a specific attribute, averaged over all the relevant events, satisfies the value average threshold assertion. • always pattern is satisfied when all the relevant events satisfy the always pattern assertion • sequence pattern is satisfied when the relevant event set contains at least one event instance for each event type in the participant set, and the order of the event instances is identical to the order of the event types in the participant set. • increasing pattern is satisfied by an attribute A if for all the relevant events, e1 << e2 e1.A < e2.A • relative max distance pattern is satisfied when the maximal distance between any two relevant events satisfies the max threshold assertion • moving toward pattern is satisfied when for any pair of relevant events e1, e2 we have e1 << e2 the location of e2 is closer to a certain object then the location of e1.
Find the five highest bids within the bid interval Pattern detection example Pattern name: Manual Assignment Preparation Pattern Type: relative N highest Context: Bid Interval Relevant event types: Delivery Bid Pattern parameter: N = 5; value = Ranking Cardinality: Single deferred Taken from the Fast Flower Delivery use case
Why do we need policies - A simple example: heavy trading scenario Why defining patterns is not that easy? Because we need to tune up the semantics • Given: • A stream of events of a single topic, about the activity in the stock market for a certain stock. • An event is produced every 10 minutes when there is trade in the stock. • Each event consists of: quote (current stock-quote), volume (an accumulated volume of traded events within these 10 minutes). • A selection specification: “trigger an automatic trade program if the volume exceeds 300,000 3 times within an hour; pass as an argument the last quote and the sum of the 3 volume values”. • How many times the trade programming is triggered ; • Which arguments are used in each triggering?
Pattern policies Evaluation policy—This determines when the matching sets are produced Deferred Cardinality policy—This determines how many matching sets are produced within a single context partition Repeated type policy—This determines what happens if the matching step encounters multiple events of the same type Every - Override – First – Last – with max/min value of…
Pattern policies – cont. Consumption policy—This specifies what happens to a participant event after it has been included in a matching set Consume Bounded reuse Reuse Order policy—This specifies how temporal order is defined By occurrence time – by detection time – by stream position – by attribute
Our entire culture is context sensitive • In the play “The Tea house of the August Moon” one of the characters says: Pornography question of geography • This says that in different geographical contexts people view things differently • Furthermore, the syntax of the language (no verbs) is typical to the way that the people of Okinawa are talking When hearing concert people are not talking, eating, and keep their mobile phone on “silent”.
Context has three distinct roles (which may be combined) The events that relate to each customer are processed separately Partition the incoming events Grouping together events that happened in the same hour at the same location Grouping events together Different processing for Different context partitions Determining the processing
Temporal Spatial State Oriented Segmentation Oriented Context Definition Acontext is a named specification of conditions that groups event instances so that they can be processed in a related way. It assigns each event instance to one or more context partitions. A context may have one or more context dimensions.
Context Types Examples Segmentation Oriented “All Children 2-5 years old” “All platinum customers” Temporal Spatial “Every day between 08:00 and 10:00 AM” “A week after borrowing a disk” “A time window bounded by TradingDayStart and TradingDayEnd events” “3 miles from the traffic accident location” “Within an authorized zone in a manufactory” Context State Oriented “Airport security level is red” “Weather is stormy”