Reliable Messaging for Grids and Web Services

1. Reliable Messaging for Grids and Web Services Geoffrey Fox, Shrideep Pallickara, Damodar Yemme, Hasan Bulut and Sima Patel (gcf, spallick, dyemme, hbulut and skpatel)@indiana.edu Community Grids Lab Indiana University

2. Message-Based Reliability • Web Services exchange messages and interact with resources that produce and absorb messages • Action and state (if exists) of a service defined by messages • Our approach to Reliability is based on a building a messaging infrastructure that is intrinsically reliable • WS-RM and WS-Reliability for web services • Naradabrokering message-oriented middleware

3. SS Database SS SS SS SS SS SS SS Raw Data  Data  Information  Knowledge  Wisdom AnotherGrid Decisions AnotherGrid SS SS SS SS FS FS OS MD MD FS Portal OS OS FS OS SOAP Messages OS FS FS FS AnotherService FS FS MD MD OS MD OS OS FS Other Service FS FS FS FS MD OS OS OS FS FS FS MD MD FS Filter Service OS AnotherGrid FS MetaData FS FS FS MD Sensor Service SS SS SS SS SS SS SS SS SS SS AnotherService

4. Applications of our Technology • Point-to-point generic linkage of services using WSRM with messages saved in databases as required in specification • Scalable Management Architecture to support dynamic collections of entities • Applied first to the brokers used in distributed messaging of NaradaBrokering • Management of the streams of data from sensors and web-cams • Allow real-time replay and annotation based on real-time saving of messages forming streams

5. WSRM • WSRM describes a protocol that facilitates the reliable delivery of messages between two web service endpoints in the presence of component, system or network failures. • WSRM facilitates the reliable delivery of messages from the source (or originator) of messages to the sink (or destination) of messages. • The delivery (and ordering) guarantees are valid over a group of messages, which is referred to as a sequence.

6. Creation of Sequences • In WSRM prior to ensuring reliable delivery of messages between the endpoints, the source initiates an exchange with the sink pertaining to the creation of a Sequence. • This Sequence is intended to facilitate the grouping of a set of related messages. • This Sequence is identified by an identifier, typically a UUID. Other information associated with the Sequence include information regarding ─ • The source and the sink • Policy information related to protocol constants such as acknowledgement and retransmission intervals. • Security related information if needed.

7. WSRM Sequences • In WSRM all messages issued by a source exist within the context of a Sequence that was established prior to communications. • Once a source has determined that all messages within a Sequence have been received at the sink, the source initiates an exchange to terminate this sequence. • The specification allows for a maximum of 264 -1 messages within a Sequence. • The specification places no limits on the number of Sequences between a specific source and sink. • However, it is expected that at any given time there is NO more than 1 active Sequence between 2 specific endpoints.

8. Publishing Messages in WSRM • Every message from the source contains two pieces of information ─ • The Sequence that this message is a part of and • A monotonically increasing Message Number within this Sequence. • These Message Numbers enable the tracking of problems, if any, in the intended message delivery at a sink. • Message Numbers enable the determination of out of order receipt of messages as well as message losses.

9. Issuing Acknowledgments • In WSRM a sink is expected to issue acknowledgements back to the source upon receipt of messages. • This acknowledgement contains information about • The Sequence and • The Message Numbers within this Sequence. • An acknowledgement must be issued only after a certain time ─ the acknowledgement interval ─ has elapsed since the receipt of the first unacknowledged message. • This acknowledgement may cover a single message or a group of messages within a Sequence.

10. Processing Acknowledgments • Upon receipt of this acknowledgement a source can determine which messages might have been lost in transit and proceed to retransmit the missed messages. • Thus if a sink has acknowledged the receipt of messages 1 ─ 10 and 13 ─ 18. • The source can conclude that messages with Message Numbers 11 and 12 were lost en route to the sink and proceed to retransmit these messages.

11. Retransmissions and Error Corrections • A source may also pro-actively initiate the retransmission of a message for which that an acknowledgement has not been received within a specified time ─ the retransmission interval ─ after which it was issued. • In WSRM error corrections can also be initiated at the sink; this is done through the use of negative acknowledgements. • Negative acknowledgments identify the message numbers that have not been received at a sink. • Message Numbers increase monotonically. If Message Numbers 1,2,3,4 and 8 within a specific Sequence have been received at a sink. • This sink can easily conclude that it has not received messages with message numbers 5,6 and 7 from the source.

12. Notification of Errors • WSRM provides for notification of errors in processing between the endpoints involved in reliable delivery. • These are routed back as SOAP Faults. • The range of errors can vary from an inability to decipher a message’s content to complex errors pertaining to violations in implied agreements between the interacting source and sink. • All errors are reported as faults with the appropriate wsa:Action attribute, and encapsulated in WSRM fault elements.

13. Comments on WSRM Implementation • We are delivering this to the UK Open Middleware Infrastructure Institute • We built WS-Eventing that is available in OMII 2.3.3 http://www.omii.ac.uk/news/newsdetail.jsp?id=25 in FINS Project • WS-RM is currently being tested in OMII container (FIRMS Project) and is expected to be finished in a month (but not released by OMII until they do further tests) • WS-RM and WS-Eventing use SOAP handlers that are not well supported in current Axis used by OMII; we should hope Axis 2 will be soon mature enough to use

15. Times are Microseconds

16. Times are Microseconds

17. NaradaBrokering Management Framework

18. NaradaBrokering 2003-2006 • Messaging infrastructure for collaboration, peer-to-peer and Grids Implements JMS and native high-performance protocols (message transit time of 1 to 2 ms per hop) • Order-preserving message transport with QoS and security profiles • Support for different underlying transport such as TCP, UDP, Multicast, RTP • SOAP message supportand WS-Eventing, WS-RM and WS-Reliability. • WS-Notification when specification agreed • Active replay support: Pause and Replay live streams. • Stream Linkage: can link permanently multiple streams – using in annotation of real-time video streams • Replicated storage support for fault tolerance and resiliency to storage failures. • Management: HPSearch Scripting Interface to streams and brokers (uses WS-Management) • Broker Topics and Message Discovery: Locate appropriate • Integration with Axis2 Web Service Container (?) • High Performance Transport supporting SOAP Infoset

19. Management of services • We prefer to build Grids (collections of web services) that use distributed publish-subscribe message-oriented middleware to transport all messages. • This is NaradaBrokering and one can bind SOAP to NB transport (very different from WS-Notification/Eventing) building a handler for this • NB will guarantee message delivery and its distributed nature has implicit reliability • However we need to maximize reliability of this infrastructure including attention to network QoS, firewalls etc.

20. Management Architecture Multiple DistributedManagee Instances With web service proxy Multiple DistributedManager Instances

21. Features of the Managee Service • The distributed managers use NaradaBrokering itself for robust messaging with the “Managees” (Web Service adaptors to each broker in NaradaBroker networker)

22. Features of the Manager Service • WS-Management used for communicating between Managers and Managees • Managers implement policy and user instructions but this very primitive

23. Generic Recording and Replay Framework

24. e-Annotation Archived Stream Annotated e-Annotation Player Player Stream Player Whiteboard Archived stream Annotation / WB e - Annotation e - Annotation player player Whiteboard Player Archived Real Time Real TimeStream List Stream List Player Real time Real time stream Archieved stream list player stream list

25. Generic Recording and Replay Framework • A generic framework for recording and replay of any type of streaming event or data. • Active replay of streams: Real-time (live) streams can be replayed, paused and rewound while streams are being recorded. • Fast forward is available for the duration of the recorded stream. • Note streams are collections of events and events are essentially messages • Rewind is same as undo (as in Office) • Replay is same as redo i.e. re-apply sequence of messages to a Web service ports

26. Generic Recording and Replay Framework • Stream linkage: Multiple streams are linked together to construct a session. • A collaboration session can be recorded and replayed within this framework. Examples; • Anabas – Uses JMS events to transport data such as whiteboard, shared display, audio, etc. • GlobalMMCS – Uses NaradaBrokering RTP Events to transport audio and video data. • Streams can be added/removed to/from session dynamically while the session is being recorded. • Maintains metadata information for recorded sessions and their streams. • Dynamic metadata stored in high performance light weight WS-Context service

27. Uniform Event Type For Generic Framework • Received events are wrapped inside NaradaBrokering native events (NBEvent) with some event specific information. • Received event is placed to the payload of the NBEvent as is to preserve original data and related information. • NBEvent also contains timestamp information to timespace original events during replay and event type to initiate appropriate player for that event type. • Events and related metadata are stored in database tables.

28. Session Recorders • Session recorder includes topic recorders that subscribe to each topic defined in that session. • Topic recorders are like subscribing clients receiving the streaming events. Topic recorders are specialized for event types. i.e. JMS events need JMS topic recorder to receive those type of events. • Event types for those streams are already known from the initiated record request.

29. Session Recorders II • Figure on the left depicts a scenario for recording of JMS event or NaradaBrokering RTPEvent producer clients

30. Session Recorders II • Session recorders construct NBEvents with necessary information, such as timestamps, in NBEvent’s header. • Along with each constructed NBEvent, a companion event is also constructed for reliable delivery mechanism. • Session recorders support initiating/stopping a record, adding/removing new streams to/from the recording session

31. Session Players • The primary purpose of session player is to simulate clients in the original session. • To achieve this; • Each recorded topic (or stream) is mapped to a new topic and events of the same original topic are released to the mapped topic. • While releasing the events, timespacing between events are preserved. • Utilizes Time Differential Service to timespace events

32. Session Players II • Figure on the left depicts a scenario for playing a session with multiple streams (NB RTPEvent or JMS event based streams)

33. Session Players II • Recording of live streams are available for replay as soon as they are stored to the reliable storage. • Session players support replay, pause, rewind and fast forward operations. When one of those operations is requested, it is applied to all of the topics (streams) in that session.

34. Time Differential Service (TDS) • Replay of events rely on one critical service: Time differential service. • Each replay session has one dedicated TDS to achieve replay, pause, rewinding and fast forwarding of the streams in the session in one operation. • Achieves synchronization of multiple streams in the same session by maintaining a shared buffer for those streams. • Maintains the timespace between the replay events equal to the timespace between the original received events . • Resolution of this timespacing is one millisecond; events can be timespaced with one millisecond accuracy.

35. Time Differential Service II • TDS can be maintained on robust node (NaradaBrokering node that provides stable storage) or on client side. • Maintaining TDS on robust node increases load of robust node because of its buffer and threads responsible for releasing the events. • It is required to support current streaming clients such as JMS clients, RTPEvent clients or other streaming clients that does not maintain an internal buffer • Maintaining TDS on client side allows more session replay support for one robust node.

36. Scalability of Generic Record/Replay Framework • Framework is scalable in the sense that it can support many clients for the same replay session. For example, • When a collaboration session (i.e. Anabas or GlobalMMCS session) is replayed, clients only need to subscribe to the replay topics. The number of clients depends on the number of clients supported by the broker, not by the robust node. • This framework can easily be extended to many other event/data types such as RealMedia, raw audio/video data or streaming text.