NaradaBrokering – Building P2P Grids

1. Access Grid meeting – August 21st, 2002. NaradaBrokering – Building P2P Grids PTLIU Laboratory for Community Grids Geoffrey Fox, Shrideep Pallickara Computer Science, Informatics, Physics Indiana University, Bloomington IN 47404http://www.naradabrokering.org http://grids.ucs.indiana.edu/ptliupages uri="http://www.naradabrokering.org" email="gcf@indiana.edu"

2. JXTA and Grids • JXTA and Grid architectures can be implemented as Web Services interacting with (XML-based) messages • We built a “Grid Messaging System” NaradaBrokering that implements generalized publish-subscribe mechanism in a network of “brokers/rendezvous peers” • NaradaBrokering can replace Java Message Service – Grid-like system • Used to run our synchronous collaboration system Garnet supporting shared display, text chats, Jabber instant messenger …. • NaradaBrokering is integrated with JXTA (as a proxy to rendezvous peers) and can provide reliable messaging between peer groups (and inside?) • We are building Collaboration (shared application and audio-video conferencing) as a Web Service • XGSP (XML General Session Protocol) is meant to include H323 SIP and (later) JXTA sessions (peer groups) • JXTA will be able to invoke Access Grid, Polycom, Shared Display sessions http://www.naradabrokering.org/ gcf, spallick@indiana.edu

3. Different Web Service Organizations • Everything is a resource implemented as a Web Service, whether it be: • back end supercomputers and a petabyte data • Microsoft PowerPoint and this file • Web Services communicate by messages ….. • Grids and Peer to Peer (P2P) networks can be integrated by building both in terms of Web Services with different (or in fact sometimes the same) implementations of core services such as registration, discovery, life-cycle, collaboration and event or message transport ….. • Gives a Peer-to-Peer Grid • NaradaBrokering is an example of Event or Message Service linking web services together http://www.naradabrokering.org/ gcf, spallick@indiana.edu

4. Database Database Event/MessageBrokers Event/MessageBrokers Integrate P2P and Grid/WS Peer to Peer Grid Peers Resource FacingWeb Service Interfaces Web Service Interfaces Peers User FacingWeb Service Interfaces A democratic organization Peer to Peer Grid http://www.naradabrokering.org/ gcf, spallick@indiana.edu

5. Role of Event/Message Brokers • We will use events and messages interchangeably • An event is a time stamped message • Our systems are built from clients, servers and “event brokers” • These are logical functions – a given computer can have one or more of these functions • In P2P networks, computers typically multifunction; in Grids one tends to have separate function computers • Event Brokers “just” provide message/event services; servers provide traditional distributed object services as Web services • There are functionalities that only depend on event itself and perhaps the data format; they do not depend on details of application and can be shared among several applications • NaradaBrokering is designed to provide these functionalities • MPI provided such functionalities for all parallel computing http://www.naradabrokering.org/ gcf, spallick@indiana.edu

6. Destination Source Matching Routing Filter workflow Web Service 1 Web Service 2 (Virtual)Queue WSDLPorts WSDLPorts Broker NaradaBrokering implements an Event Web Service • Filter is mapping to PDA or slow communication channel (universal access) – see our PDA adaptor • Workflow implements message process • Routing illustrated by JXTA • Destination-Source matching illustrated by JMS using Publish-Subscribe mechanism http://www.naradabrokering.org/ gcf, spallick@indiana.edu

7. Engineering Issues Addressedby Event / Messaging Service • Application level Quality of Service – give audio highest priority • Tunnel through firewalls • Filter messages to slow (collaborative or real time) clients • Hardware multicast is erratically implemented (Event service can dynamically use software multicast) • Scaling of software multicast • Elegant implementation of Collaboration in a Groove Networks (done better) style • Integrate synchronous and asynchronous collaboration http://www.naradabrokering.org/ gcf, spallick@indiana.edu

8. Features of Event Service I • Based on a network of cooperating broker nodes • Cluster based architecture allows system to scale to arbitrary size. • Messages are not sent directly from P to S but rather from P to Broker B and from Broker B to subscriber S • Synchronous systems: B acts as a real-time router/filterer • Messages can be archived and software multicast • Asynchronous systems: B acts as an XML database and workflow engine • Subscription is in each case, roughly equivalent to a database query • Aims at millisecond latencies (MPI aims at microsecond latencies) http://www.naradabrokering.org/ gcf, spallick@indiana.edu

9. Features of Event Web Service II • In principle Message brokering can be virtual and compiled away in the same way that WSDL ports can be bound in real time to optimal transport mechanism • All Web Services are specified in XML but can be implemented quite differently • Audio Video Conferencing sessions could be negotiated using SOAP (raw XML) messages and agree to use certain video codecs transmitted by UDP/RTP • There is a collection of XML Schema – call it GXOS – specifying event service and requirements of message streams and their endpoints • One can sometimes compile message streams specified in GXOS to MPI or to local method call • Event Service must support dynamic heterogeneous protocols http://www.naradabrokering.org/ gcf, spallick@indiana.edu

10. Features of Event Web Service III • The event web service is naturally implemented as a dynamic distributed network • Required for fault tolerance and performance • A new classroom joins my online lecture • A broker is created to handle students – multicast locally my messages to classroom; handle with high performance local messages between students • Company X sets up a firewall • The event service sets up brokers either side of firewall to optimize transport through the firewall • Note all message based applications use same message service • Web services imply ALL applications are (possibly virtual) message based http://www.naradabrokering.org/ gcf, spallick@indiana.edu

11. JMS Compliance • Features: • JMS clients written while conforming to spec • JMS clients are vendor agnostic • JMS providers do not interoperate with each other • Rationale • Providing support for JMS clients within the system. • Mature specification with several existing applications • Opens NaradaBrokering to these applications • Bring NaradaBrokering functionality to JMS systems • Distributed solution, load balancing, scaling and failure resiliency. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

12. Providing JMS support • JMS Interactions • Supported locally or mapped to corresponding NaradaBrokering calls • JMS Interconnection Bridge • Operations supported locally or mapped to corresponding NaradaBrokering interactions • One bridge instance per connection • Maintains list of registered sessions • Responsible for routing events to appropriate sessions • Support for • Creation of different message types e.g. ObjectMessage, BytesMessage etc. • Operations that can be invoked on these message types. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

13. Providing JMS Support • Topic • NaradaBrokering topics are created as <tag,value> pairs. • JMS Topics are generally “/” separated. • JMS selector mechanism • We augment NaradaBrokering’s topic matching with the selector mechanism implemented by openJMS. • JMS subscriptions • Mapped to corresponding NaradaBrokering subscription requests. • The Narada JMS Event. • Encapsulates the entire JMS message as a payload for the event. • Matching is done based on the topic name contained in the message. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

14. Towards a distributed JMS solution • Features in NaradaBrokering best exploited in distributed settings. • Clients of distributed solution to inherit NaradaBrokering features • Routing efficiencies, load balancing, scaling etc. • Eliminate the Single point of failure model • Highly Available System • Existing systems should easily be replaced with distributed system http://www.naradabrokering.org/ gcf, spallick@indiana.edu

15. Broker Locators: Distributed JMS Solution • Primary Function • Discovery of brokers that clients can connect to • Obviates need for client to keep track of broker states within the broker network. • Keeps track of • Broker additions and removals • Changes to network fabric • Published Limit on concurrent connections at a broker node • Set during broker initializations • Active connections at a broker node. • Individual brokers notify changes to broker locator. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

16. Broker Locators: Features & Constraints • Features • Dynamic Real time Load Balancing • Connection requests forked to best available broker. • Incorporation of new brokers into solution • A newly added broker is among the best brokers to handle a connection request. • Slower clients could all be hosted on specific brokers • Eliminates broker choking resulting from servicing very slow clients. • Constraints • Availability (multiple per domain) • Should not constitute a bottleneck or single point of failure. • Minimal logic • Should not affect processing pertaining to any node in the system. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

17. Broker Locator: Decision Metrics • IP-address of requesting client • Published limit on concurrent connections • Number of active connections still possible • Availability of broker • A simple ping test • If broker is not available, remove broker from list of available brokers. • Computing capabilities at a broker • CPU speed, RAM etc. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

18. Broker Locator: Sequence of Operations • Locate valid broker • Propagate broker information to client • Hostname/IP-address information • Port number on which it listens for connections • Transport protocol over which it communicates • Client then uses info to establish communication channel with broker • Done transparently. • Clients with multiple connections • A client could sometimes have connections to multiple brokers. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

19. JMS Performance Data • SonicMQ (version 3.0) and NaradaBrokering broker • Dual CPU (Pentium 3, 1 GHz, 256 MB RAM) machine. • 100 subscribers • Over 10 different JMSTopicConnections • All hosted on a Dual CPU (Pentium 3, 866 MHz, 256 MB RAM) machine. • Publisher and Measuring Subscriber • Hosted on another dual CPU (Pentium 3, 866 MHz, 256 MB RAM) machine. • Operating System and Run time Environment • Linux (version 2.2.16) • Java 2 JRE (Java 1.3.1, Blackdown-FCS, mixed-mode) http://www.naradabrokering.org/ gcf, spallick@indiana.edu

20. Performance Data:Factors Measured • Transit Delay • No need for clock synchronization and accounting for clock drifts. • Standard Deviation in the transit delay for the sample of messages received. • System Throughput • Measured in terms of rate at which messages are received. • Factors measured under varying • publish rates • message payload sizes. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

21. Performance: Transit Delay & Standard Deviation Lower Payloads & Low Publish Rates http://www.naradabrokering.org/ gcf, spallick@indiana.edu

22. Performance: Transit Delay & Standard Deviation Higher Payloads & Low Publish Rates http://www.naradabrokering.org/ gcf, spallick@indiana.edu

23. Performance: Transit Delay & Standard Deviation Lower Payloads & High Publish Rates http://www.naradabrokering.org/ gcf, spallick@indiana.edu

24. Performance: System Throughput Lower Payloads & Higher Publish Rates http://www.naradabrokering.org/ gcf, spallick@indiana.edu

25. Why P2P • Core features • Resource Sharing & Discovery • CPU cycles: SETI@home, Folding@HOME • File Sharing: Napster, Gnutella • Deployments user driven • No dedicated management • Management of resources • Expose resources & specify security strategy • Replicate resources based on demand • Dynamic peer groups, fluid memberships • Sophisticated search mechanisms • Peers respond based on their interpretations • Responses do not conform to traditional templates. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

26. What are the downsides? • Interactions are attenuated • Localized • Fragmented world of multiple P2P subsystems • Routing not very sophisticated • Inefficient network utilization (Tragedy of Commons) • Simple forwarding • Peer Traces (to eliminate echoing) • Attenuations (to suppress propagation) http://www.naradabrokering.org/ gcf, spallick@indiana.edu

27. JXTA • Set of protocols to support P2P interactions. • Planned bridges to (and from) other P2P systems. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

28. NaradaBrokering & JXTA Interaction Model • Based on proxy model • Acts as both Rendezvous peer (JXTA routers) and NaradaBrokering client. • No changes to JXTA core or straitjacketing of interactions • Change made to Rendezvous layer • Peers are not aware that they interact with a Narada-JXTA proxy or Rendezvous peer. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

29. What we plan to accomplish • Better routing and network utilization • Bridge isolated P2P subsystems • Efficiently locate/replicate shared resources • Since interactions are routed through the broker network, brokers can build efficient resource maps. • Incorporate GXOS support in NaradaBrokering • Then use JXTA search to locate resources stored in the distributed XML database. • Provide the notion of reliable interactions (for peers attached to Narada-JXTA proxies) • Grid/WebServices generated dynamically when complex tasks are initiated could be managed by peer groups. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

30. Narada-JXTA Proxy • Glean relevant information from JXTA interactions. • Peer group advertisements • Requests/Responses to be part of peer group. • Messages sent to a peer group. • Subscribe to relevant topics to ensure delivery • Construct corresponding Narada-JXTA event from interactions. http://www.naradabrokering.org/ gcf, spallick@indiana.edu

31. Applications • Integrated NaradaBrokering-JXTA environment tested under JXTA shell and myJxta (InstantP2P) • Plan to integrate myJxta into Anabas with NaradaBrokering managing P2P and middle-tier (JMS) style interactions. • JXTA-Jabber interface via NaradaBrokering/Anabas. • (Jabber, PDA, Anabas) linkage to be combined with (Jxta, NaradaBrokering, Anabas linkage). http://www.naradabrokering.org/ gcf, spallick@indiana.edu

32. PDA Collaboration Event Filter GMS =JMS orNarada http://www.naradabrokering.org/ gcf, spallick@indiana.edu

33. R R R U U U WSViewer WSDisplay F F F F F F I I I I I I WebService WebService WebService O O O O O O WS Viewer WS Display WS Viewer WSDisplay Shared Input Port (Replicated WS) Collaboration Collaboration as a WSSet up Session Master Event(Message)Service OtherParticipants http://www.naradabrokering.org/ gcf, spallick@indiana.edu

34. WSDL R U Application orContent source F F WSViewer WSDisplay I I O O Web Service Shared Output Port Collaboration Collaboration as a WSSet up Session Web Service Message Interceptor Master WS Viewer WS Display Event(Message)Service OtherParticipants WS Viewer WSDisplay http://www.naradabrokering.org/ gcf, spallick@indiana.edu

35. Web Service Architecturefor Audio Video Conferencing http://www.naradabrokering.org/ gcf, spallick@indiana.edu

36. XGSP: Introduction • Registration Method registration server with its alias name and current location • Session Command Method Membership Control Commands, Session Control Commands • Query Method discover various properties about the system • Session Channel Binding Method bind the RTP channels of a client into the media server http://www.naradabrokering.org/ gcf, spallick@indiana.edu

37. XGSP: Example <SessionDes> <SessionName> PervasiveTech Seminar </SessionName> <SessionID> 1234567 </SessionID> <SessionCreator> Ahmet@indiana.edu </SessionCreator> <SessionInfo> this is a meeting on the XGSP </SessionInfo> <SessionPlace> Lobby Room </SessionPlace> <SessionTime> <StartTime> (EastTime) 10:00AM </StartTime> <EndTime> (EastTime) 12:00AM </EndTime> </SessionTime> <SessionURI> http://grids.ucs.indiana.edu/~ag </SessionURI> <SessionParticipants> <Participant> Wenjun@156.56.103.129 </Participant> <Participant> Hasan@156.56.103.27 </Participant> <Participant> Shrideeper@156.56.103.111 </Participant> </SessionParticipants> <ContactInfo> wewu@indiana.edu </ContactInfo> </SessionDes> http://www.naradabrokering.org/ gcf, spallick@indiana.edu

38. NaradaBrokering Futures • Higher Performance – reduce minimum transit time to around one millisecond • Substantial operational testing • Security – allow Grid (Kerberos/PKI) security mechanisms • Support of more protocols with dynamic switching as in JXTA – SOAP, RMI, RTP/UDP • Have prototype RTP/UDP support • Integration of simple XML database model using JXTA Search to manage distributed archives • More formal specification of “native mode” and dynamic instantiation of brokers • General Collaborative Web services http://www.naradabrokering.org/ gcf, spallick@indiana.edu