Complex event processing seminar hkust september 2004
1 / 58

Complex event processing Seminar : HKUST – September 2004 - PowerPoint PPT Presentation

  • Uploaded on
  • Presentation posted in: General

Complex event processing Seminar : HKUST – September 2004. Opher Etzion IBM Research Laboratory in Haifa As stated in the Gartner report, apples are …. Wow, Gartner! Perhaps this wouldn’t be a total waste of time after all. YES!! It works every time.

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.

Download Presentation

Complex event processing Seminar : HKUST – September 2004

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript

Complex event processing seminar hkust september 2004

Complex event processingSeminar : HKUST – September 2004

Opher Etzion

IBM Research Laboratory in Haifa

Complex event processing seminar hkust september 2004

As stated in the Gartner report, apples are ….

Wow, Gartner!

Perhaps this wouldn’t be a total waste of timeafter all

YES!! It works every time.

The Gartner quote always gets their attention.

Active behavior scenario 1

Active Behavior – scenario 1

Your Refrigerator called me..

Active behavior scenario 2

Active Behavior - Scenario 2

At least N people of my team are in this building now

Active technologies scenario 3

Active technologies – scenario 3

  • An information about a certain target arrived from two different sources within the last hour AND a contradictory information did not arrive within the last hour, and there have been at least five active sources in the same area within the last hour then handle the target.

Active behavior scenario 4

Active behavior : Scenario 4

  • Whenever a traffic congestion occurs, re-evaluate the traffic-lights timing policies and change it.

What is the common denominator

What is the common denominator ?

  • All of these are event-driven

  • They are not driven by a single event, so some processing of the events is needed.

  • We can have some hints about the types of processing needed in each case…

What is the talk about

What is the talk about ?

  • Events --- what are they ?

  • How do they relate to the rest of the universe ?

  • What are the types of event processes ?

  • AMIT – a CEP example

  • Other examples

  • Some use cases

  • Some research and pragmatic challenges

What is an event

What is an event ?

An event is a significant (in some context) instantaneous (happens in a specific point in time) atomic (happens completely or not at all) occurrence

Are those events ?

Event collection middleware

Event collection middleware

  • A platform to communicate events

  • Event sources: application, workflow, database, IT resource – needs instrumentation and normalization.

  • Event transfer: publish/subscribe (with content).

  • Many technical issues:

    • Scalability

    • Standard protocols

    • Event store

    • Event bus

What is the connection between events and data

What is the connection between events and data ?

  • Database does not have an inherent semantics, it can represent everything (including events)

  • But – semantically there is a distinction between:

    • State: snapshot of the state of (the appropriate subset of) the universe at a certain point in time.

    • Transition: transfer from state to state.

  • Database processing (e.g. SQL) is state processing.

  • So – does event represent transitions ?

    • Sometimes.

      • A transition is an event.

      • Not all events change states that are of interest.

Is event processing different from data processing

Is event processing different from data processing ?

  • In database processing – a snapshot that includes multiple entities is processed. All the processing is a function of values that exists within this snapshot.

  • In event processing --- the history of events is processed, and can also relate to the state information. Temporal processing is dominant.

  • Event processing subsumes data processing (but usually not all the power of SQL is needed).

  • Event processing can be expressed in SQL (the data-stream people are working on it) but in many cases it is not easy (exercise: write in SQL query that finds if a sequence of at least 8 events of 8 different event types has occurred in a sequence within an hour anytime).

Complex event processing a model based approach









Complex event processing – a model based approach

What is an event model

What is an event model ?

  • Data model does not talk only about data, but also about things in the world and dependencies among them (e.g ER, semantic data model).

  • Event model is an extended data model in which events play role as a first class citizen.

  • Events are connected to :

    • Other events

    • Context (what’s that ???)

    • Messages

    • Entities

    • Databases or other state keepers

    • Flows

Event processing situation

Event processing - Situation

  • Situation is defined as a transition in the universe that requires reaction (either “reactive” situation or “proactive” situation).

  • One of the main event processing goals is situation detection

    • An approximation for situation is an inferred causality event within a context, the function can optionally contain other players (data, state etc..) S = F (e1,…en, context, [state information]).

    • This approximation equals the situation when the function is deterministic; in other cases we shall need to operate uncertainty measures.

  • Other types of event processing relate to relationships of events with everything else.

Complex event processing seminar hkust september 2004

Situation detection

Some examples from various domains

Some examples from various domains

  • Alert if the IBM stock has gone up in 3 percent within two hours, and the Dow Jones did not go up in more than 1 percent at the same period.

  • Alert if three memory problems occurred during the last hour.

  • Alert if the same request was reassigned to three agents, and no answer was given to the requester.

Relationships among events

Relationships among events

  • Causality:

    • Observed causality : two events that has statistical correlation that indicates that one of them is an antecedent of the other (example: getting out of the car, locking the car).

    • Inferred causality: An event that is signaled as a result of processing in which the other event participated in (example: traffic jam identified, traffic-lights policy re-calculated).

  • Subset hierarchy :

    • possibly conditional generalization/specialization relations (printer problem, hardware problem)

  • Cross-section :

    • Events with certain conditions are considered as other virtual events (e.g. all events that relate to the same object).



  • Event is instantaneous – occurs in a certain period of time.

  • Sometimes event processing is done within a context

  • Examples:

    • Within the working hours

    • The last 30 minutes of trade

    • within 60 minutes from the time that the request was sent

    • From transaction start to transaction end

  • Context has :

    • Validity interval.

    • possibly multiple instances by some partition criterion (e.g. by customer).

    • possibly spatial characteristics.

  • Events are related to contexts:

    • Start and end of contexts are events.

    • Event processing can be a function of context.

Context awareness

Context Awareness

  • Composite perspectives of the environment

    • Semantic perspective designates environment information about a specific object or entity (e.g. users that are members of the same group).

    • Temporal perspective designates environment information within a specific temporal element (e.g. network overload in one hour).

    • Spatial perspective designates environment information within a specific location or area (e.g. vehicles near a traffic problem).

    • State perspective designates environment information within a specific state (e.g. low market volume).

Basic text slide

Temporal context

Temporal Context

  • Designate a collection of events that

    • occur within a temporal interval

    • Bounded by initiator and terminator

    • Has maximal length, initiation and termination policies

    • Multiple lifespan may exists simultaneously











Basic text slide

Spatial context

Spatial context

  • Express the spatial perspective of Amit context

  • Designate a collection of events that are originated from the same region

  • Either a circle specified by a center coordinate and a radius or a polygon specified by a set of coordinates.

  • Fixed or moving



Basic text slide

Events and messages

Events and Messages

  • There is common misconception that events and messages are the same.

  • In fact there are some connections:

    • A message may represent a description of event occurrence – this is one of the common ways to report events.

    • There are events associated with messages (created, sent, received, acknowledged…) like any other entity.

    • Complex event processing is part of message brokering/mediation.

Events and entities

Events and entities

  • An event can refer to one or more entities,

  • Example: John sell the bike to Jim (type of event sale, entities: John, bike, Jim with different roles).

  • An entity has a role in an event (seller, buyer, merchandise)

  • An event may have a role in the entity (starting, ending, disrupting – can be interpreted as change the entity status)

Events and databases again

Events and databases – again

  • An event may (but does not have to) change state (or results in a database update)

  • A database operation occurrence is an event

  • Event processing can look at both events and states.

Events and activities and flows

Events and activities (and flows…)

  • Event/situation can trigger activities

  • Activity can signal events

  • A flow can contain activities, event, data and messages with all interactions.

  • In case of a flow --- an event/situation can add/modify/delete sub-flows dynamically and can interrupt running flows

  • Transition in flow states are events.

Complex event processing seminar hkust september 2004

  • In early days of programming languages it was used for interrupts/exception handling

  • Real-time applications raised requirements for reaction based on time.

  • Active databases started in the late 1980-ies

  • System and network management tools emerged in the early 1990-ies

  • Publish/Subscribe system appeared in the lat 1990-ies

  • Currently: monitoring, management application, business process integration, straight-through-processing…Gartner’s : CEP, BAM, RTE.

Short history of event processing ..

Active behavior specification

Active behavior specification:

  • Embedded in regular programming languages.

  • Rule-based approaches (reactive/proactive programming):

    • Condition-Action rules (event is hidden)

    • Event-Condition-Action rules

    • Situation-[Condition]- Action rules

  • Model based approaches (reflective programming)

    • Semantic net approach.

    • Self-stabilization approach (data-driven).

  • The “action” can be

    a “business rule”

Complex event processing seminar hkust september 2004

Event-Condition-Action rules

  • Making the processing event driven

  • The paradigm :

    • when event occurred

    • if a condition is satisfied

    • perform action

  • A variation can be E(CA)*

  • Example: When a message about Microsoft stock quote has arrived, If the value is less than 25, then notify all customers in the subscription-list.

Complex event processing seminar hkust september 2004

Situation based processing

  • Next step in the evolution – from event based to situation based.

  • The concept of situation is what triggers the action from the user’s point of view (may not be a single event)

  • This is an abstraction over the universe of transitions in the same way that SQL query or view is an abstraction over the universe of states

  • It is roughly equivalent to the term “composite event within a context”, but may have uncertainty associated with it.

Complex event processing seminar hkust september 2004

Situation examples

  • A client has withdrawn at least 3 times a sum of more than $100,000 that he deposited at most 2 days before the withdrawal [looking for money exceptional movement]

  • A client has withdrawn within 2 hours from two ATM machines that are more then 200 KM apart [fraud detection]

  • A client wishes to be notified when IBM stock is up more than 3 percent if he is in the office [personalized location-aware publish/subscribe]

  • At the end of the day at least 2% of the orders have not completed [monitoring]

Complex event processing seminar hkust september 2004

  • Publish/subscribe is : event - action.

  • Publish/subscribe with filtering is: event - condition – action (current state of the art).

  • Situation based publish/subscribe is the next generation – enable “personalization of push technology”.

Relationship to

Publish/Subscribe Technology

Relationship to event correlation

Relationship to event correlation

  • The term “event correlation” has originated from network management.

  • The idea is to find event that have statistical correlation among them (and hence the name) and possibly filter out some of them to cope with “event storm”

  • It has some notion of causality (usually – two events occurring together within a fixed time interval).

  • Used in system management applications, and as an event filtering tool.

Complex event processing seminar hkust september 2004


Active Middleware Technology

Amit technologies context diagram

Exception Handling Metadata

Amit – Technologies Context diagram














Create new Events



Change Definitions

Amit Metadata

(Definition Manager)

External (pub/sub)



ADI Model











Complex event processing seminar hkust september 2004


(all, sequence)



(atleast, atmost, nth)





(every, after, at)


(not, unless)





(retain, override,...)





Repeat Mode

(always, once)

(immediate, delayed, deferred)

Detection Mode






The situation concept

Amit tooling today

Amit Tooling Today

Selected additional complex event processing solutions

Selected additional complex event processing solutions

  • Apama

  • Elity

  • Actimize

  • Ispheres

Some applications

Some applications

of Complex Event Processing

Transaction flow monitoring

Transaction Flow Monitoring

Transaction Level Alerts:

  • Acknowledgements is not consistent with the sending order.

  • 15 minutes before a settlement has to close, a message is waiting to be handled by a compliance officer due to insufficient credit.

  • No acknowledgement/reject received from the stock exchange within one hour from sending a message.

    Business Level Alerts:

  • Abnormal number of payments received from a specific bank (account) within the first two hours or business

  • Three rejects received within a single working day, for FED messages of the same platinum client.

Finance scenarios

Finance Scenarios

Trade Regulation Breach

CEPS verifies that sells and subsequent purchases of large quantities of stock meet regulatory requirements

Credit Breach

CEPS initially allows a credit limit to be exceeded (by no more than 10%). The second time the limit is exceeded, CEPS routes the order to a credit officer for approval

Trade Execution Delay

Notify if a purchase order was sent for processing and no response was received within the time specified by the SLA.

Fraud DetectionReport when multiple credit card purchases are performed within an hour or (any given time frame) at a distance greater than 300 km (or any given distance).


On demand ecrm architecture and dataflow example

On Demand eCRM Architecture and Dataflow Example:

  • Database stores all historical customer information

  • A CEP server receives relevant customer data from database when customer enters the store

  • Each event is processed when customer is in the store

  • Action managers performs actions – recommendation of complementary products, sales on frequently purchased items etc..


Data Warehouse


Action Manager


Wireless server



Complex event processing seminar hkust september 2004

Security Scenario

Homeland security


Unusual activity


Insurance scenario

Insurance Scenario

CEP helps improve claim processing by automating key activities such as:

Identifying invalid claims

Detection of potential fraudulent claims

We can help release delayed payments according to pre-specified conditions.

Identifying problematic health insurance providers that do not comply with regulations.


Higher level technologies that use cep

Higher level technologies that use CEP

  • The “sense and respond” loop

  • Real-time analytics

  • Autonomic computing

On demand control loop sense and respond

On-demand control loop (sense and respond) :



Real time analytics

Real-time analytics

  • Departure from traditional use of analytic tools that may not be time constrained.

  • The traffic light problem is an example

  • Other examples:

    • Re-calculation of network configuration policies when part of the network is disabled (e.g. due to “denial of service attack”.

    • Re-establishing of queue priorities policies

  • Trade-off between time and quality of solution (cannot get to the optimal solution in 1 minutes, how should I get to the best possible solution given these time constraints)..

Autonomic computing

Alerts, events and problem analysis request interface

SLA/Policy interface, interprets and translates into "control logic"

Autonomic computing





Policy Interpreter

Analysis Engines

Policy Validations

Policy Transforms

Policy Resolution


Plan Generators


Rules Engines

Workflow Engine



Service Dispatcher

Simple Correlators



Scheduler Engine

Metric Managers

Recent Activity Log


Distribution Engine

Interface to real and virtualized resources and components that regulate control.

Additional research topics

Additional research topics

  • Real-time aspects

  • Distribution and parallelism

  • Transactional support

  • Temporal issues

  • Uncertainty in complex event processing.

  • Software engineering aspects.

Real time aspects

Real-time aspects:

  • The end-to-end process involving the CEP may have real-time constraints.

  • This may inflict real-time Real-time awareness built-in operations:

    • Scheduling

    • Prioritization

    • Relevance of “late” events.

Distribution and parallelism

Distribution and Parallelism

  • For scalability reasons:

    • The ability to apply N engines and balance the processing load

    • Note, that there can be many nested situations, thus full partition may not be possible

    • Establishing minimal traffic among the various engines.

    • Reference: M. Shmueli, O. Etzion - Parallel Implementation of Composite Events. ICDCS Workshops 2002: 579-580

  • For high availability:

    • Requires support in clustering and failover.

    • Usually done by basing on middleware services

Transactional support

Transactional support

  • May require non-ACID transactional support :

    • Total rollback may not be possible, since we cannot say that the “event did not happen”, even if we fail to process its consequences.

    • Events that are part of the process may belong to different transactions, only the last of them “closes the loop”.

    • However, some of the process can be “atomic”.

Temporal issues

Temporal issues

  • Events may not arrive in the same order they are produced

  • The time-stamps on different events may not be consistent with relative timings.

  • There may be a communication delay to report events

    • How long should we wait for an event ?

    • What happens we get event beyond this time-out ?

  • In general – how do we process:

    • Retroactive events (events about the past)

    • Predictive events (“certain” events about the future)

Uncertainty aspects

Uncertainty aspects :

  • There are several sources for uncertainty:

    • Uncertainty that the event has (or has not) occurred

    • Uncertainty in the details of the events itself

    • Uncertainty that the SITUATION is equivalent to this specific function of events and contexts

    • Uncertainty that the right context is identified,

Software engineering aspects

Software Engineering aspects:

  • A programming paradigm that deserves:

    • Methodologies - as a programming tool, and as part of a bigger picture

    • Modeling tools, automatic creations of rules

    • Debugging --- all the known difficulties of rule debugging --- interactions among situations, halting problem, determinism and sequencing.

Future stand alone vs embedded technology

Future --- stand-alone vs. embedded technology

  • Complex event processing is being developed in two main contexts:

    • As a central service collecting events from various sources (will be part of all middleware products)..

    • As an embedded technology inside other frameworks/products/solutions (seem to grow more rapidly)

  • According to Gartner’s hype-cycle, time to full maturity is 5-7 years.



  • D. Luckham – The power of events: An Introduction to Complex Event Processing. Addison-Wesley, 2002

  • A. Adi, O. Etzion – Amit, the situation manager. VLDB Journal, 13(2), 173-203, 2004.

  • Gartner reports:

    • “Events will transform application servers”

    • “Hype cycle for application integration middleware and platforms 2003”

  • Vendors URLs :





  • Foundations :

    • Chakravarthy-S, and Mishra-D. "Snoop: an expressive event specification language for active databases." Data and Knowledge Engineering 14.1 (1994): 1-26.

    • Tombros-D, Geppert-A, and Dittrich-KR. "Semantics of Reactive Components in Event-Driven Workflow Execution". CAiSE 1997: 409-422.

    • Yemini-SA, Kliger-S, Mozes-E, Yemini-Y, and Ohsie-D. "High speed and robust event correlation." IEEE Communications Magazine. 34.5 (1996): 82-90.

    • Zimmer-D, and Unland-R. "A General Model for Specification of the Semantics of Complex Events in Active Database Management Systems." C-LAB Report. 1998.

    • Zimmermann-J, and Buchmann-A. "REACH." Active Rules in Database Systems. Springer Verlag, 1999. 263-277.


  • Login