Complex event processing Seminar : HKUST – September 2004

1. Complex event processingSeminar : HKUST – September 2004 Opher Etzion IBM Research Laboratory in Haifa opher@il.ibm.com

2. As stated in the Gartner report, apples are …. Wow, Gartner! Perhaps this wouldn’t be a total waste of timeafter all YES!! It works every time. The Gartner quote always gets their attention.

3. Active Behavior – scenario 1 Your Refrigerator called me..

4. Active Behavior - Scenario 2 At least N people of my team are in this building now

5. Active technologies – scenario 3 • An information about a certain target arrived from two different sources within the last hour AND a contradictory information did not arrive within the last hour, and there have been at least five active sources in the same area within the last hour then handle the target.

6. Active behavior : Scenario 4 • Whenever a traffic congestion occurs, re-evaluate the traffic-lights timing policies and change it.

7. What is the common denominator ? • All of these are event-driven • They are not driven by a single event, so some processing of the events is needed. • We can have some hints about the types of processing needed in each case…

8. What is the talk about ? • Events --- what are they ? • How do they relate to the rest of the universe ? • What are the types of event processes ? • AMIT – a CEP example • Other examples • Some use cases • Some research and pragmatic challenges

9. What is an event ? An event is a significant (in some context) instantaneous (happens in a specific point in time) atomic (happens completely or not at all) occurrence Are those events ?

10. Event collection middleware • A platform to communicate events • Event sources: application, workflow, database, IT resource – needs instrumentation and normalization. • Event transfer: publish/subscribe (with content). • Many technical issues: • Scalability • Standard protocols • Event store • Event bus

11. What is the connection between events and data ? • Database does not have an inherent semantics, it can represent everything (including events) • But – semantically there is a distinction between: • State: snapshot of the state of (the appropriate subset of) the universe at a certain point in time. • Transition: transfer from state to state. • Database processing (e.g. SQL) is state processing. • So – does event represent transitions ? • Sometimes. • A transition is an event. • Not all events change states that are of interest.

12. Is event processing different from data processing ? • In database processing – a snapshot that includes multiple entities is processed. All the processing is a function of values that exists within this snapshot. • In event processing --- the history of events is processed, and can also relate to the state information. Temporal processing is dominant. • Event processing subsumes data processing (but usually not all the power of SQL is needed). • Event processing can be expressed in SQL (the data-stream people are working on it) but in many cases it is not easy (exercise: write in SQL query that finds if a sequence of at least 8 events of 8 different event types has occurred in a sequence within an hour anytime).

13. entity context state activity event situation data flow Complex event processing – a model based approach

14. What is an event model ? • Data model does not talk only about data, but also about things in the world and dependencies among them (e.g ER, semantic data model). • Event model is an extended data model in which events play role as a first class citizen. • Events are connected to : • Other events • Context (what’s that ???) • Messages • Entities • Databases or other state keepers • Flows

15. Event processing - Situation • Situation is defined as a transition in the universe that requires reaction (either “reactive” situation or “proactive” situation). • One of the main event processing goals is situation detection • An approximation for situation is an inferred causality event within a context, the function can optionally contain other players (data, state etc..) S = F (e1,…en, context, [state information]). • This approximation equals the situation when the function is deterministic; in other cases we shall need to operate uncertainty measures. • Other types of event processing relate to relationships of events with everything else.

16. Situation detection

17. Some examples from various domains • Alert if the IBM stock has gone up in 3 percent within two hours, and the Dow Jones did not go up in more than 1 percent at the same period. • Alert if three memory problems occurred during the last hour. • Alert if the same request was reassigned to three agents, and no answer was given to the requester.

18. Relationships among events • Causality: • Observed causality : two events that has statistical correlation that indicates that one of them is an antecedent of the other (example: getting out of the car, locking the car). • Inferred causality: An event that is signaled as a result of processing in which the other event participated in (example: traffic jam identified, traffic-lights policy re-calculated). • Subset hierarchy : • possibly conditional generalization/specialization relations (printer problem, hardware problem) • Cross-section : • Events with certain conditions are considered as other virtual events (e.g. all events that relate to the same object).

19. Context • Event is instantaneous – occurs in a certain period of time. • Sometimes event processing is done within a context • Examples: • Within the working hours • The last 30 minutes of trade • within 60 minutes from the time that the request was sent • From transaction start to transaction end • Context has : • Validity interval. • possibly multiple instances by some partition criterion (e.g. by customer). • possibly spatial characteristics. • Events are related to contexts: • Start and end of contexts are events. • Event processing can be a function of context.

20. Context Awareness • Composite perspectives of the environment • Semantic perspective designates environment information about a specific object or entity (e.g. users that are members of the same group). • Temporal perspective designates environment information within a specific temporal element (e.g. network overload in one hour). • Spatial perspective designates environment information within a specific location or area (e.g. vehicles near a traffic problem). • State perspective designates environment information within a specific state (e.g. low market volume). Basic text slide

21. Temporal Context • Designate a collection of events that • occur within a temporal interval • Bounded by initiator and terminator • Has maximal length, initiation and termination policies • Multiple lifespan may exists simultaneously busy busy free free free dysfunctional free busy free free Basic text slide

22. Spatial context • Express the spatial perspective of Amit context • Designate a collection of events that are originated from the same region • Either a circle specified by a center coordinate and a radius or a polygon specified by a set of coordinates. • Fixed or moving dysfunctional dysfunctional Basic text slide

23. Events and Messages • There is common misconception that events and messages are the same. • In fact there are some connections: • A message may represent a description of event occurrence – this is one of the common ways to report events. • There are events associated with messages (created, sent, received, acknowledged…) like any other entity. • Complex event processing is part of message brokering/mediation.

24. Events and entities • An event can refer to one or more entities, • Example: John sell the bike to Jim (type of event sale, entities: John, bike, Jim with different roles). • An entity has a role in an event (seller, buyer, merchandise) • An event may have a role in the entity (starting, ending, disrupting – can be interpreted as change the entity status)

25. Events and databases – again • An event may (but does not have to) change state (or results in a database update) • A database operation occurrence is an event • Event processing can look at both events and states.

26. Events and activities (and flows…) • Event/situation can trigger activities • Activity can signal events • A flow can contain activities, event, data and messages with all interactions. • In case of a flow --- an event/situation can add/modify/delete sub-flows dynamically and can interrupt running flows • Transition in flow states are events.

27. In early days of programming languages it was used for interrupts/exception handling • Real-time applications raised requirements for reaction based on time. • Active databases started in the late 1980-ies • System and network management tools emerged in the early 1990-ies • Publish/Subscribe system appeared in the lat 1990-ies • Currently: monitoring, management application, business process integration, straight-through-processing…Gartner’s : CEP, BAM, RTE. Short history of event processing ..

28. Active behavior specification: • Embedded in regular programming languages. • Rule-based approaches (reactive/proactive programming): • Condition-Action rules (event is hidden) • Event-Condition-Action rules • Situation-[Condition]- Action rules • Model based approaches (reflective programming) • Semantic net approach. • Self-stabilization approach (data-driven). • The “action” can be a “business rule”

29. Event-Condition-Action rules • Making the processing event driven • The paradigm : • when event occurred • if a condition is satisfied • perform action • A variation can be E(CA)* • Example: When a message about Microsoft stock quote has arrived, If the value is less than 25, then notify all customers in the subscription-list.

30. Situation based processing • Next step in the evolution – from event based to situation based. • The concept of situation is what triggers the action from the user’s point of view (may not be a single event) • This is an abstraction over the universe of transitions in the same way that SQL query or view is an abstraction over the universe of states • It is roughly equivalent to the term “composite event within a context”, but may have uncertainty associated with it.

31. Situation examples • A client has withdrawn at least 3 times a sum of more than $100,000 that he deposited at most 2 days before the withdrawal [looking for money exceptional movement] • A client has withdrawn within 2 hours from two ATM machines that are more then 200 KM apart [fraud detection] • A client wishes to be notified when IBM stock is up more than 3 percent if he is in the office [personalized location-aware publish/subscribe] • At the end of the day at least 2% of the orders have not completed [monitoring]

32. Publish/subscribe is : event - action. • Publish/subscribe with filtering is: event - condition – action (current state of the art). • Situation based publish/subscribe is the next generation – enable “personalization of push technology”. Relationship to Publish/Subscribe Technology

33. Relationship to event correlation • The term “event correlation” has originated from network management. • The idea is to find event that have statistical correlation among them (and hence the name) and possibly filter out some of them to cope with “event storm” • It has some notion of causality (usually – two events occurring together within a fixed time interval). • Used in system management applications, and as an event filtering tool.

34. Amit Active Middleware Technology

35. Exception Handling Metadata Amit – Technologies Context diagram Events Instances Amit Tooling Exceptions GUI Exception Handling Rule-based Exceptions Detections Model-based tools Create new Events Action Manager Change Definitions Amit Metadata (Definition Manager) External (pub/sub) Situation Manager ADI Model Events Definitions Rule-based Situations Conclusions Alerts Exceptions Templates Rules ARAD

36. JOINING (all, sequence) e2 COUNTING (atleast, atmost, nth) Operators Events e1 TEMPORAL (every, after, at) 3 (not, unless) ABSENCE e3 Attributes Conditions (retain, override,...) (where...) e1.id Key e2.name e3.key Situation Repeat Mode (always, once) (immediate, delayed, deferred) Detection Mode Context e5 e8 Terminator Initiator The situation concept

37. Amit Tooling Today

39. Selected additional complex event processing solutions • Apama • Elity • Actimize • Ispheres • …

40. Some applications of Complex Event Processing

41. Transaction Flow Monitoring Transaction Level Alerts: • Acknowledgements is not consistent with the sending order. • 15 minutes before a settlement has to close, a message is waiting to be handled by a compliance officer due to insufficient credit. • No acknowledgement/reject received from the stock exchange within one hour from sending a message. Business Level Alerts: • Abnormal number of payments received from a specific bank (account) within the first two hours or business • Three rejects received within a single working day, for FED messages of the same platinum client.

42. Finance Scenarios Trade Regulation Breach CEPS verifies that sells and subsequent purchases of large quantities of stock meet regulatory requirements Credit Breach CEPS initially allows a credit limit to be exceeded (by no more than 10%). The second time the limit is exceeded, CEPS routes the order to a credit officer for approval Trade Execution Delay Notify if a purchase order was sent for processing and no response was received within the time specified by the SLA. Fraud DetectionReport when multiple credit card purchases are performed within an hour or (any given time frame) at a distance greater than 300 km (or any given distance). Finance

43. On Demand eCRM Architecture and Dataflow Example: • Database stores all historical customer information • A CEP server receives relevant customer data from database when customer enters the store • Each event is processed when customer is in the store • Action managers performs actions – recommendation of complementary products, sales on frequently purchased items etc.. CRM Data Warehouse Server Action Manager CEP SERVER Wireless server Hand-Held Retail

44. Security Scenario Homeland security CEP Unusual activity Security

45. Insurance Scenario CEP helps improve claim processing by automating key activities such as: Identifying invalid claims Detection of potential fraudulent claims We can help release delayed payments according to pre-specified conditions. Identifying problematic health insurance providers that do not comply with regulations. Insurance

46. Higher level technologies that use CEP • The “sense and respond” loop • Real-time analytics • Autonomic computing

47. On-demand control loop (sense and respond) : 06n AFS -

48. Real-time analytics • Departure from traditional use of analytic tools that may not be time constrained. • The traffic light problem is an example • Other examples: • Re-calculation of network configuration policies when part of the network is disabled (e.g. due to “denial of service attack”. • Re-establishing of queue priorities policies • Trade-off between time and quality of solution (cannot get to the optimal solution in 1 minutes, how should I get to the best possible solution given these time constraints)..

49. Alerts, events and problem analysis request interface SLA/Policy interface, interprets and translates into "control logic" Autonomic computing Sensors Effectors Plan Analyze Policy Interpreter Analysis Engines Policy Validations Policy Transforms Policy Resolution Execute Plan Generators Monitor Rules Engines Workflow Engine Knowledge Filters Service Dispatcher Simple Correlators Topology Calendar Scheduler Engine Metric Managers Recent Activity Log Policy Distribution Engine Interface to real and virtualized resources and components that regulate control.

50. Additional research topics • Real-time aspects • Distribution and parallelism • Transactional support • Temporal issues • Uncertainty in complex event processing. • Software engineering aspects.