Complex event processing seminar hkust september 2004
Download
1 / 58

Complex event processing Seminar : HKUST – September 2004 - PowerPoint PPT Presentation


  • 109 Views
  • Uploaded on

Complex event processing Seminar : HKUST – September 2004. Opher Etzion IBM Research Laboratory in Haifa [email protected] As stated in the Gartner report, apples are …. Wow, Gartner! Perhaps this wouldn’t be a total waste of time after all. YES!! It works every time.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Complex event processing Seminar : HKUST – September 2004' - zelia


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Complex event processing seminar hkust september 2004

Complex event processingSeminar : HKUST – September 2004

Opher Etzion

IBM Research Laboratory in Haifa

[email protected]


As stated in the Gartner report, apples are ….

Wow, Gartner!

Perhaps this wouldn’t be a total waste of timeafter all

YES!! It works every time.

The Gartner quote always gets their attention.


Active behavior scenario 1
Active Behavior – scenario 1

Your Refrigerator called me..


Active behavior scenario 2
Active Behavior - Scenario 2

At least N people of my team are in this building now


Active technologies scenario 3
Active technologies – scenario 3

  • An information about a certain target arrived from two different sources within the last hour AND a contradictory information did not arrive within the last hour, and there have been at least five active sources in the same area within the last hour then handle the target.


Active behavior scenario 4
Active behavior : Scenario 4

  • Whenever a traffic congestion occurs, re-evaluate the traffic-lights timing policies and change it.


What is the common denominator
What is the common denominator ?

  • All of these are event-driven

  • They are not driven by a single event, so some processing of the events is needed.

  • We can have some hints about the types of processing needed in each case…


What is the talk about
What is the talk about ?

  • Events --- what are they ?

  • How do they relate to the rest of the universe ?

  • What are the types of event processes ?

  • AMIT – a CEP example

  • Other examples

  • Some use cases

  • Some research and pragmatic challenges


What is an event
What is an event ?

An event is a significant (in some context) instantaneous (happens in a specific point in time) atomic (happens completely or not at all) occurrence

Are those events ?


Event collection middleware
Event collection middleware

  • A platform to communicate events

  • Event sources: application, workflow, database, IT resource – needs instrumentation and normalization.

  • Event transfer: publish/subscribe (with content).

  • Many technical issues:

    • Scalability

    • Standard protocols

    • Event store

    • Event bus


What is the connection between events and data
What is the connection between events and data ?

  • Database does not have an inherent semantics, it can represent everything (including events)

  • But – semantically there is a distinction between:

    • State: snapshot of the state of (the appropriate subset of) the universe at a certain point in time.

    • Transition: transfer from state to state.

  • Database processing (e.g. SQL) is state processing.

  • So – does event represent transitions ?

    • Sometimes.

      • A transition is an event.

      • Not all events change states that are of interest.


Is event processing different from data processing
Is event processing different from data processing ?

  • In database processing – a snapshot that includes multiple entities is processed. All the processing is a function of values that exists within this snapshot.

  • In event processing --- the history of events is processed, and can also relate to the state information. Temporal processing is dominant.

  • Event processing subsumes data processing (but usually not all the power of SQL is needed).

  • Event processing can be expressed in SQL (the data-stream people are working on it) but in many cases it is not easy (exercise: write in SQL query that finds if a sequence of at least 8 events of 8 different event types has occurred in a sequence within an hour anytime).


Complex event processing a model based approach

entity

context

state

activity

event

situation

data

flow

Complex event processing – a model based approach


What is an event model
What is an event model ?

  • Data model does not talk only about data, but also about things in the world and dependencies among them (e.g ER, semantic data model).

  • Event model is an extended data model in which events play role as a first class citizen.

  • Events are connected to :

    • Other events

    • Context (what’s that ???)

    • Messages

    • Entities

    • Databases or other state keepers

    • Flows


Event processing situation
Event processing - Situation

  • Situation is defined as a transition in the universe that requires reaction (either “reactive” situation or “proactive” situation).

  • One of the main event processing goals is situation detection

    • An approximation for situation is an inferred causality event within a context, the function can optionally contain other players (data, state etc..) S = F (e1,…en, context, [state information]).

    • This approximation equals the situation when the function is deterministic; in other cases we shall need to operate uncertainty measures.

  • Other types of event processing relate to relationships of events with everything else.



Some examples from various domains
Some examples from various domains

  • Alert if the IBM stock has gone up in 3 percent within two hours, and the Dow Jones did not go up in more than 1 percent at the same period.

  • Alert if three memory problems occurred during the last hour.

  • Alert if the same request was reassigned to three agents, and no answer was given to the requester.


Relationships among events
Relationships among events

  • Causality:

    • Observed causality : two events that has statistical correlation that indicates that one of them is an antecedent of the other (example: getting out of the car, locking the car).

    • Inferred causality: An event that is signaled as a result of processing in which the other event participated in (example: traffic jam identified, traffic-lights policy re-calculated).

  • Subset hierarchy :

    • possibly conditional generalization/specialization relations (printer problem, hardware problem)

  • Cross-section :

    • Events with certain conditions are considered as other virtual events (e.g. all events that relate to the same object).


Context
Context

  • Event is instantaneous – occurs in a certain period of time.

  • Sometimes event processing is done within a context

  • Examples:

    • Within the working hours

    • The last 30 minutes of trade

    • within 60 minutes from the time that the request was sent

    • From transaction start to transaction end

  • Context has :

    • Validity interval.

    • possibly multiple instances by some partition criterion (e.g. by customer).

    • possibly spatial characteristics.

  • Events are related to contexts:

    • Start and end of contexts are events.

    • Event processing can be a function of context.


Context awareness
Context Awareness

  • Composite perspectives of the environment

    • Semantic perspective designates environment information about a specific object or entity (e.g. users that are members of the same group).

    • Temporal perspective designates environment information within a specific temporal element (e.g. network overload in one hour).

    • Spatial perspective designates environment information within a specific location or area (e.g. vehicles near a traffic problem).

    • State perspective designates environment information within a specific state (e.g. low market volume).

Basic text slide


Temporal context
Temporal Context

  • Designate a collection of events that

    • occur within a temporal interval

    • Bounded by initiator and terminator

    • Has maximal length, initiation and termination policies

    • Multiple lifespan may exists simultaneously

busy

busy

free

free

free

dysfunctional

free

busy

free

free

Basic text slide


Spatial context
Spatial context

  • Express the spatial perspective of Amit context

  • Designate a collection of events that are originated from the same region

  • Either a circle specified by a center coordinate and a radius or a polygon specified by a set of coordinates.

  • Fixed or moving

dysfunctional

dysfunctional

Basic text slide


Events and messages
Events and Messages

  • There is common misconception that events and messages are the same.

  • In fact there are some connections:

    • A message may represent a description of event occurrence – this is one of the common ways to report events.

    • There are events associated with messages (created, sent, received, acknowledged…) like any other entity.

    • Complex event processing is part of message brokering/mediation.


Events and entities
Events and entities

  • An event can refer to one or more entities,

  • Example: John sell the bike to Jim (type of event sale, entities: John, bike, Jim with different roles).

  • An entity has a role in an event (seller, buyer, merchandise)

  • An event may have a role in the entity (starting, ending, disrupting – can be interpreted as change the entity status)


Events and databases again
Events and databases – again

  • An event may (but does not have to) change state (or results in a database update)

  • A database operation occurrence is an event

  • Event processing can look at both events and states.


Events and activities and flows
Events and activities (and flows…)

  • Event/situation can trigger activities

  • Activity can signal events

  • A flow can contain activities, event, data and messages with all interactions.

  • In case of a flow --- an event/situation can add/modify/delete sub-flows dynamically and can interrupt running flows

  • Transition in flow states are events.


  • In early days of programming languages it was used for interrupts/exception handling

  • Real-time applications raised requirements for reaction based on time.

  • Active databases started in the late 1980-ies

  • System and network management tools emerged in the early 1990-ies

  • Publish/Subscribe system appeared in the lat 1990-ies

  • Currently: monitoring, management application, business process integration, straight-through-processing…Gartner’s : CEP, BAM, RTE.

Short history of event processing ..


Active behavior specification
Active behavior specification: interrupts/exception handling

  • Embedded in regular programming languages.

  • Rule-based approaches (reactive/proactive programming):

    • Condition-Action rules (event is hidden)

    • Event-Condition-Action rules

    • Situation-[Condition]- Action rules

  • Model based approaches (reflective programming)

    • Semantic net approach.

    • Self-stabilization approach (data-driven).

  • The “action” can be

    a “business rule”


Event-Condition-Action rules interrupts/exception handling

  • Making the processing event driven

  • The paradigm :

    • when event occurred

    • if a condition is satisfied

    • perform action

  • A variation can be E(CA)*

  • Example: When a message about Microsoft stock quote has arrived, If the value is less than 25, then notify all customers in the subscription-list.


Situation based processing interrupts/exception handling

  • Next step in the evolution – from event based to situation based.

  • The concept of situation is what triggers the action from the user’s point of view (may not be a single event)

  • This is an abstraction over the universe of transitions in the same way that SQL query or view is an abstraction over the universe of states

  • It is roughly equivalent to the term “composite event within a context”, but may have uncertainty associated with it.


Situation examples interrupts/exception handling

  • A client has withdrawn at least 3 times a sum of more than $100,000 that he deposited at most 2 days before the withdrawal [looking for money exceptional movement]

  • A client has withdrawn within 2 hours from two ATM machines that are more then 200 KM apart [fraud detection]

  • A client wishes to be notified when IBM stock is up more than 3 percent if he is in the office [personalized location-aware publish/subscribe]

  • At the end of the day at least 2% of the orders have not completed [monitoring]


  • Publish/subscribe is : event - action. interrupts/exception handling

  • Publish/subscribe with filtering is: event - condition – action (current state of the art).

  • Situation based publish/subscribe is the next generation – enable “personalization of push technology”.

Relationship to

Publish/Subscribe Technology


Relationship to event correlation
Relationship to event correlation interrupts/exception handling

  • The term “event correlation” has originated from network management.

  • The idea is to find event that have statistical correlation among them (and hence the name) and possibly filter out some of them to cope with “event storm”

  • It has some notion of causality (usually – two events occurring together within a fixed time interval).

  • Used in system management applications, and as an event filtering tool.


Amit interrupts/exception handling

Active Middleware Technology


Amit technologies context diagram

Exception Handling Metadata interrupts/exception handling

Amit – Technologies Context diagram

Events

Instances

Amit

Tooling

Exceptions

GUI

Exception

Handling

Rule-based

Exceptions

Detections

Model-based

tools

Create new Events

Action

Manager

Change Definitions

Amit Metadata

(Definition Manager)

External (pub/sub)

Situation

Manager

ADI Model

Events

Definitions

Rule-based

Situations

Conclusions

Alerts

Exceptions

Templates

Rules

ARAD


JOINING interrupts/exception handling

(all, sequence)

e2

COUNTING

(atleast, atmost, nth)

Operators

Events

e1

TEMPORAL

(every, after, at)

3

(not, unless)

ABSENCE

e3

Attributes

Conditions

(retain, override,...)

(where...)

e1.id

Key

e2.name

e3.key

Situation

Repeat Mode

(always, once)

(immediate, delayed, deferred)

Detection Mode

Context

e5

e8

Terminator

Initiator

The situation concept


Amit tooling today
Amit Tooling Today interrupts/exception handling


Selected additional complex event processing solutions
Selected additional complex event processing solutions interrupts/exception handling

  • Apama

  • Elity

  • Actimize

  • Ispheres


Some applications

Some applications interrupts/exception handling

of Complex Event Processing


Transaction flow monitoring
Transaction Flow Monitoring interrupts/exception handling

Transaction Level Alerts:

  • Acknowledgements is not consistent with the sending order.

  • 15 minutes before a settlement has to close, a message is waiting to be handled by a compliance officer due to insufficient credit.

  • No acknowledgement/reject received from the stock exchange within one hour from sending a message.

    Business Level Alerts:

  • Abnormal number of payments received from a specific bank (account) within the first two hours or business

  • Three rejects received within a single working day, for FED messages of the same platinum client.


Finance scenarios
Finance Scenarios interrupts/exception handling

Trade Regulation Breach

CEPS verifies that sells and subsequent purchases of large quantities of stock meet regulatory requirements

Credit Breach

CEPS initially allows a credit limit to be exceeded (by no more than 10%). The second time the limit is exceeded, CEPS routes the order to a credit officer for approval

Trade Execution Delay

Notify if a purchase order was sent for processing and no response was received within the time specified by the SLA.

Fraud DetectionReport when multiple credit card purchases are performed within an hour or (any given time frame) at a distance greater than 300 km (or any given distance).

Finance


On demand ecrm architecture and dataflow example
On Demand eCRM Architecture and Dataflow Example: interrupts/exception handling

  • Database stores all historical customer information

  • A CEP server receives relevant customer data from database when customer enters the store

  • Each event is processed when customer is in the store

  • Action managers performs actions – recommendation of complementary products, sales on frequently purchased items etc..

CRM

Data Warehouse

Server

Action Manager

CEP SERVER

Wireless server

Hand-Held

Retail


Security Scenario interrupts/exception handling

Homeland security

CEP

Unusual activity

Security


Insurance scenario
Insurance Scenario interrupts/exception handling

CEP helps improve claim processing by automating key activities such as:

Identifying invalid claims

Detection of potential fraudulent claims

We can help release delayed payments according to pre-specified conditions.

Identifying problematic health insurance providers that do not comply with regulations.

Insurance


Higher level technologies that use cep
Higher level technologies that use CEP interrupts/exception handling

  • The “sense and respond” loop

  • Real-time analytics

  • Autonomic computing


On demand control loop sense and respond
On-demand control loop (sense and respond) : interrupts/exception handling

06n

AFS -


Real time analytics
Real-time analytics interrupts/exception handling

  • Departure from traditional use of analytic tools that may not be time constrained.

  • The traffic light problem is an example

  • Other examples:

    • Re-calculation of network configuration policies when part of the network is disabled (e.g. due to “denial of service attack”.

    • Re-establishing of queue priorities policies

  • Trade-off between time and quality of solution (cannot get to the optimal solution in 1 minutes, how should I get to the best possible solution given these time constraints)..


Autonomic computing

Alerts, events and problem analysis request interface interrupts/exception handling

SLA/Policy interface, interprets and translates into "control logic"

Autonomic computing

Sensors

Effectors

Plan

Analyze

Policy Interpreter

Analysis Engines

Policy Validations

Policy Transforms

Policy Resolution

Execute

Plan Generators

Monitor

Rules Engines

Workflow Engine

Knowledge

Filters

Service Dispatcher

Simple Correlators

Topology

Calendar

Scheduler Engine

Metric Managers

Recent Activity Log

Policy

Distribution Engine

Interface to real and virtualized resources and components that regulate control.


Additional research topics
Additional research topics interrupts/exception handling

  • Real-time aspects

  • Distribution and parallelism

  • Transactional support

  • Temporal issues

  • Uncertainty in complex event processing.

  • Software engineering aspects.


Real time aspects
Real-time aspects: interrupts/exception handling

  • The end-to-end process involving the CEP may have real-time constraints.

  • This may inflict real-time Real-time awareness built-in operations:

    • Scheduling

    • Prioritization

    • Relevance of “late” events.


Distribution and parallelism
Distribution and Parallelism interrupts/exception handling

  • For scalability reasons:

    • The ability to apply N engines and balance the processing load

    • Note, that there can be many nested situations, thus full partition may not be possible

    • Establishing minimal traffic among the various engines.

    • Reference: M. Shmueli, O. Etzion - Parallel Implementation of Composite Events. ICDCS Workshops 2002: 579-580

  • For high availability:

    • Requires support in clustering and failover.

    • Usually done by basing on middleware services


Transactional support
Transactional support interrupts/exception handling

  • May require non-ACID transactional support :

    • Total rollback may not be possible, since we cannot say that the “event did not happen”, even if we fail to process its consequences.

    • Events that are part of the process may belong to different transactions, only the last of them “closes the loop”.

    • However, some of the process can be “atomic”.


Temporal issues
Temporal issues interrupts/exception handling

  • Events may not arrive in the same order they are produced

  • The time-stamps on different events may not be consistent with relative timings.

  • There may be a communication delay to report events

    • How long should we wait for an event ?

    • What happens we get event beyond this time-out ?

  • In general – how do we process:

    • Retroactive events (events about the past)

    • Predictive events (“certain” events about the future)


Uncertainty aspects
Uncertainty aspects : interrupts/exception handling

  • There are several sources for uncertainty:

    • Uncertainty that the event has (or has not) occurred

    • Uncertainty in the details of the events itself

    • Uncertainty that the SITUATION is equivalent to this specific function of events and contexts

    • Uncertainty that the right context is identified,


Software engineering aspects
Software Engineering aspects: interrupts/exception handling

  • A programming paradigm that deserves:

    • Methodologies - as a programming tool, and as part of a bigger picture

    • Modeling tools, automatic creations of rules

    • Debugging --- all the known difficulties of rule debugging --- interactions among situations, halting problem, determinism and sequencing.


Future stand alone vs embedded technology
Future --- stand-alone vs. embedded technology interrupts/exception handling

  • Complex event processing is being developed in two main contexts:

    • As a central service collecting events from various sources (will be part of all middleware products)..

    • As an embedded technology inside other frameworks/products/solutions (seem to grow more rapidly)

  • According to Gartner’s hype-cycle, time to full maturity is 5-7 years.


References
References interrupts/exception handling

  • D. Luckham – The power of events: An Introduction to Complex Event Processing. Addison-Wesley, 2002

  • A. Adi, O. Etzion – Amit, the situation manager. VLDB Journal, 13(2), 173-203, 2004.

  • Gartner reports:

    • “Events will transform application servers”

    • “Hype cycle for application integration middleware and platforms 2003”

  • Vendors URLs :

    • http://www.apama.com

    • http://www.elity.com

    • http://www.actimize.com

    • http://www.ispheres.com

  • Foundations :

    • Chakravarthy-S, and Mishra-D. "Snoop: an expressive event specification language for active databases." Data and Knowledge Engineering 14.1 (1994): 1-26.

    • Tombros-D, Geppert-A, and Dittrich-KR. "Semantics of Reactive Components in Event-Driven Workflow Execution". CAiSE 1997: 409-422.

    • Yemini-SA, Kliger-S, Mozes-E, Yemini-Y, and Ohsie-D. "High speed and robust event correlation." IEEE Communications Magazine. 34.5 (1996): 82-90.

    • Zimmer-D, and Unland-R. "A General Model for Specification of the Semantics of Complex Events in Active Database Management Systems." C-LAB Report. 1998.

    • Zimmermann-J, and Buchmann-A. "REACH." Active Rules in Database Systems. Springer Verlag, 1999. 263-277.

      .


ad