Peer to Peer Grids and Collaboration

1. Peer to Peer Grids and Collaboration Euresco Conference at Castelvecchio Pascoli Italy June 16-21 2001 EuroConference on Problem-Solving Environments for Numerical Mathematics, Science and Engineering Applications Geoffrey Fox IPCRES Laboratory for Grid Technology Computer Science, Informatics, Physics Indiana University Bloomington IN fox@csit.fsu.edu p2pgrideuropejune01

2. Some Possible Truths • Technology is rapidly changing and current projects (both software and architecture) are ephemeral • Some things like CORBA and Jini may die • Grids will become P2P Networks • Java may become C# (hope not) • XML will become more important – especially for objects not just documents • Semantic Web good concept • Technologies are disruptive and cannot be ignored • Industry not academia developing most important base technologies • Not so interesting to set standards – not at MPI stage p2pgrideuropejune01

3. Classic Grid Architecture Database Database Resources Neos Composition Middle TierBrokers Service Providers Netsolve Security Portal Portal Clients Users and Devices p2pgrideuropejune01

4. Peer to Peer Grids • A Computational Grid can be defined as a collection of computers, on-line instruments, data archives and networks that are all tied together by a shared set of services which, when taken together, provide users with transparent access through interface devices to the entire set of resources. • Initial focus to a power users accessing major resources • A P2P Network can be defined as a collection of users, interface devices, computers, on-line instruments, data archives and networks that are all tied together by a shared set of services which, when taken together, provide everything transparent access to everything else. • Initial focus to building communities and access to commodity resources such as MP3 files p2pgrideuropejune01

5. Peer to Peer Network User User User Service Service Service User User User Resource Resource Resource Service Service Service Routing Routing Routing Resource Resource Resource Routing Routing Routing Peers Peers are Jacks of all Trades linked to “all” peers in community p2pgrideuropejune01

6. Peer to Peer Grid User User User Service Service Service User User User Resource Resource Resource Service Service Service Routing Routing Routing Resource Resource Resource Routing Routing Routing GMS Routing Services DynamicMessage or EventRouting fromPeers orServers GMS or GES is Grid Message/Event Service p2pgrideuropejune01

7. .antinet Database JavaBusinessLogic Discovery EnterpriseJavabeans XML Objects Castor XML > Java • This is emerging approach to Internet Systems by everybody except Microsoft New distributed database? Billions of …… FS XML objects Distributed XMLFile systems Or Databases Oracle 9i p2pgrideuropejune01

8. Collaboratory Applications • Distance Education including advanced seminars and training • Help Desk including • Microsoft helping user debug problem on home PC (connected to Internet) • Computer center consulting staff interacting over distance in real time with a user with a program bug • Yahoo staff asking in depth questions from users browsing either their knowledge or Shopping sites • Scientists brainstormingdifficult research issues in distributed locations • Virtual communities around the world from children chatting to each other or integration of distributed organizations (like ARL) • Indian Nation remaining in their homeland but participating electronically in modern economy (digital.indigineousworld.org) • Implementing next round of PET activities • Crisis Management and Command and Control for Military • Peer to Peer Networks p2pgrideuropejune01

9. What Issues are we trying to address? • A framework to build custom collaboratories • Education and Training – the Virtual University • Custom Shared Objects – not just shared PowerPoint but shared 3D visualizations (for collaborative visualization and gaming school) • Science and Engineering Research • Integrate Portals (WebCT) and Collaboration (WebEx) as both built around accessing and sharing objects • We build laboratories (collaboratories) – not desks (conventional portals) • Universal Access – CAVE, PC, Settop Box, PDA, Braille Machine • Separate specification and rendering of objects p2pgrideuropejune01

10. Goals and Features of Garnet • Integrate synchronous and asynchronous collaboration • Use same (Java) publish/subscribe Message Service to support both forms of collaboration • Integrate concept of a portal (web interfaces to applications) with collaboration • Use same XML object metadata specification GXOS to manage and share objects • Supports universal access including PDA’s collaborating with desktops • PDA Control of device or PDA alternative kiosk access gotten as a collaborative session between device and PDA • Object metadata includes rendering information for different devices p2pgrideuropejune01

11. Different Ways of Sharing • All forms of Collaboration are Event based • Different modes: Display, Export, “Event” correspond to events generated at different places in object realization/rendering pipeline • Shared Display – Events contain updates to frame buffer • Shared “Event” – Events contain updates to state of either original or transformed object • Shared Dynamic Export – Convert (rendering of) object to some standard form that is more flexible than bitmap of Shared Display. Build a custom sharing for this exported form • WebEx uses “patented sharing of virtual printer” which is equivalent to sharing export to PDF • I like shared HTML (web pages)orSVG described later p2pgrideuropejune01

12. Important Capabilities in Initial Garnet • Standard stuff: built in shared display, whiteboard, HearMeAudiocontrol, quizzes, annotations, chat/IM(Jabber.org) • Desktop video will be special case of shared display • Record and replay all features of session (SMIL) • A/V, Presentation, Annotations, Text Chat • Several Specialized Collaborative Shared Export Viewers: JSP, HTML, Acrobat .. • Initial SVG(Scalable Vector Graphics) Shared Batik Viewer • 2D Scientific Visualization/Whiteboard • Macromedia (Flash~SVG) andAdobe (already “all” to SVG) • Initial source of SVG: Convert PowerPoint VML/WMFtoSVG • Gives shared export model for PowerPoint with each client able to scale independently at high resolution p2pgrideuropejune01

13. Garnet Architecture Underpinnings • Most software written in Java (EJB); all data structures defined in XML GXOS • Unified Event Model – assume all systems communicate with XML based messages (possibly wrapped); universal rendering – portalML • All entities – client and server devices, users, programs -- defined in XML – resourceML • All events archived to enable fault tolerance and replay • GMS -- Garnet(Grid?) Message Service -- extends JMS to XML not text properties to allow server side intelligence • Dynamic Server Clusters p2pgrideuropejune01

14. Publish/Subscribe Collaboration • Integrate Asynchronous and Synchronous Collaboration • The web is full of objects – web pages sitting on web servers – and these support asynchronous collaboration • You post a web page and I later look at it in my own time • Replacing web document by a “CGI script” or servlet (web interface to program, database etc.) gives general multi-tier object sharing • This is Publish/Subscribe mechanism • If add some mechanism (automatic email or word of mouth) to tell viewing client when new information is posted • We build on JMS (Java Message Service) as Industry standard for publish/subscribe systems • Synchronous Collaboration provides “real-time” notification and automatic update of changed objects • Is JMSfast enough to do real-time? need “HP”JMS p2pgrideuropejune01

15. Everything communicates by Messages • What have we learnt from parallel computing, clusters and the Internet? • Message Passing Rules … • Note an event is just a message with (an optional) time stamp and typically (but not necessarily) an active (interrupt) message processing paradigm • Message passing enables modularity, fault tolerance, distribution • MPI for Parallel Computing – low latency etc. • JMS (or better “GMS” Grid Message Service”) high functionality p2pgrideuropejune01

16. JMS (Java Message Service) Structure in Garnet Basic primitive is a topic/property labeled queue = JXTA Pipe JMS Global (distributed) Event Receptor (Queue) Subscribe Subscribe Publish HHMS Convert Eventsto JMS HHMS (Hand Held Message Service) Optimized for Performance. JavaScript Java C++ ….. p2pgrideuropejune01

17. Performance of Commercial JMS I Non-persistent asWe do databasebackup outside JMS p2pgrideuropejune01

18. Performance of Commercial JMS II e Non-durable asour databasecopes with lateclients p2pgrideuropejune01

19. Proposed GMS Model for Messages Database • All message publication labels and subscription profiles are defined in XML Subscribes to all events to getpersistence SubscriberProfile Objects Specify Query to Event Label Message Queue Labeled by (XML)Topic Object Subscribers Publishers p2pgrideuropejune01

20. Peer to Peer P2P Networks • (Synchronous) Collaboration is critical service in P2P networks • Publish/Subscribe is mechanism we use to establish who gets what information • Gnutella and JXTA are different implementations (from JMS) of P2P information propagation • GMS can be built on top of JXTA or JMS architecture • JXTA like MyXoS identifies the implicit distributed operating system • Both have message queues as primitives • Both have Shell • Both use XML based messages • JXTA Advertisements are similar to GXOS metaobjects p2pgrideuropejune01

21. Single Server P2P Illusion Data base JMS/GMSServer p2pgrideuropejune01

22. Multiple Server P2P Illusion Data base JMS Server JMS Server Generate “Automatically” JMS Server p2pgrideuropejune01

23. P2P Grid Event Delivery Service • Dynamic Collection of some billion computers each of which can either generate, route or consume events • Publisher labels events by an (XML) object which is at simplest a URI but in general a collection of tag-values • Subscribers issue some sort of XML Query e.g. deliver all p2pgrid://garnet/Education/Graduate/ComputerScience/Syracuse/Spring2001/CPS616/Lecture3/* • Need Secure, High Performance, Efficient (don’t propagate events further than they need), Fault Tolerant delivery service • Shrideep Pallickara PhD June 1 2001 p2pgrideuropejune01

24. Some Results – 22 Servers Servers arelogically butnot necessarilyphysically distinct from clients p2pgrideuropejune01

25. Match Rates of 10% & Server Hop to client = 1 p2pgrideuropejune01

26. Match Rates of 50% & Server Hop to client = 4 p2pgrideuropejune01

27. Integration of Hand Held Clients Shared ExportPDF/HTML/ SVG Viewer Part ofJabber IM JMS/GMSServer Adaptor XML Processor Shared Display Processor GMS Processor p2pgrideuropejune01

28. Integration of Hand Held Clients • Client Device (machine) Profile stored in GXOS specifies O/S, default Screen Size modified by user (person) preferences • Dynamically updated with connection bandwidth, user updates • Application Profile stored in GXOS and modified by event stream specifies data delivered by GMS • Adaptor (Personal Server) looks like a conventional client to GMS and adapts data to specified client/user specifications • If PDA “small”, then SVG viewer on “adaptor” and ship framebuffer to PDA • “Resize” on PDA handled by adaptor, scrolling by PDA etc. • Adaptor can process complex XML queries p2pgrideuropejune01

29. PowerPoint Shared DisplayPC to PDA p2pgrideuropejune01

30. Instant Messenger Sharing PC to PDA p2pgrideuropejune01

31. SVG Sharing PC to PDA Batik Viewer on PC PowerPoint can be converted to SVGvia Illustrator or Web export p2pgrideuropejune01

32. MetaObject Event-based Computing Paradigm • All entities are MetaObjects defined in universal GXOS XML Syntax – Meta means GXOS doesn’t really want to manage Object, just information required to find, render and share it • Rendering includes Palm devices as well as PC’s • Entities are people, computers, data sources, information from e-mail, scientific visualization to digitized Bible • All actions including object changes are events – all events are GXOS objects • e-mail, paging etc. are all GXOS events uniformly routed/archived etc. • All action is instantiated or recorded in messages which are events themselves • There is an MetaObject Shell MyXoSwith basic Services (copy, create, collaborate etc.) p2pgrideuropejune01

33. A Lesson from UNIX, Windows, Yahoo … • Structured information is always labeled as a hierarchy with symbolic links e.g. xxx://A(home=root)/B/C/D …. We havesearch engines for unstructured information Symbolic Links p2pgrideuropejune01

34. Confirmed by ADL and W3C DOM • DoD’s advanced distributed learning effort (and IMS more academic consortium) use a tree structure for educational material in SCORM • http://www.adlnet.org • Add educational specific tags like prerequisitesand completion requirements • Node of tree is calledblock • ADL have an educational hierarchy with extensions for the rest of the world • Rather one should have a general tree with extensions for documents, people, education, computers, events,transactions …. • W3C DOM is at times a document object model but it is really “general object instance” looked at as a tree • UNIX and Windows file systems also hierarchical p2pgrideuropejune01

35. XML DTD Structure for Block Element Internal Node Leaf Node p2pgrideuropejune01

36. Structured and Unstructured Data • Two important hierarchies – both labeled by URI • Categories for Structured metadata – GXOS resources are labeled with these (GNDI) • Locations for Structured and unstructured dataprotocol://root/x/y/ … CORBA, Java SOAP .. link stored in metadata • Xlink/Xpath can identify an object component (value of an element) by hierarchical external location (URL) or hierarchical internal name (URI) ) • Unstructured data discovered by “Web Search” • Structured data by Category look-up or by “Web Search” • GXOS like P2P Networks (e.g. Napster) labels objects with metadata (why it is useful) not just cryptic name as in filesystems p2pgrideuropejune01

37. GXOS GNDI Structure I • Typically GXOS does NOT hold the object but rather all the metadata for object including where to find it • internalND (XML/EJB)defines where the metadata is (GNDI Garnet Naming/Directory) • externalND where the object is (JNDI Java Naming/Directory Interface or CORBA or COM or SOAP) • Idea used in WebFlow/Gateway portals – implies efficient as use natural high performance mechanisms for “real objects” • GXOS uses classic UNIX style directory structure such as //Education/University/FSU/CS/PhD/Course/Lecture • Structure says can handle quadrillion meta-objects//Jefferson/HallD/Run1/May132005/Tape5/Event1234/Detector3 … • Events are a special type of GXOS object defined with GMS Extension to basic hierarchical object p2pgrideuropejune01

38. GXOS GNDI Structure II • The more you know about an object, the easier it is to build a collaborative system • It is hard to share a general program as properties are not explicit • You put in GXOS metadata precisely those properties that need to be shared • All GXOS objects can be containers as well as having full object characteristics • Unlike UNIX where directories (only containers) and files (never explicit containers) are different • Can create a link (internalND pointer) with a profile to define special properties of this object in a new context • Intelligent symbolic links • One links User, Client Device to VE (Virtual Environment) with XML profiles mapping user to this session --> Universal Access p2pgrideuropejune01

39. Overall Structure of GXOS for a MegaMeeting EventArchive Capabilities Global Root Users Devices Documents Admin Multimedia Have a hierarchy of MegaMeetings(any collection of meetings)Course, Degree .. Are MegaMeetings MegaMeeting Meeting Meeting Meeting gxos://Education/University/FSU/CS/PhD/Course/Lecture p2pgrideuropejune01

40. Structure of GXOS For a Collaborative Session VEa VEa VEa VEb VEb VEb VEc VEc VEc Meeting An ApplicationSupporting GMS GNDI Links Users Devices GNDI Links Sharedlets Capabilities Sharedlet1 Sharedlet3 Sharedlet2 VE is VirtualEnvironment p2pgrideuropejune01

41. Semantic Web: URI and Topics etc. • The URI jxtapipe://fred/jim/agatha is equivalent to • <label objecttype=“topic” queuetype=“jxtapipe” field1=“fred” field2=“jim” field3=“agatha” /> • The <label /> syntax is more general • URI labeling allows consistency with W3C Semantic Web and indeed current practice • Everything is then a “resource” in RDF resource is a labeled object p2pgrideuropejune01

42. RDF Examples in GXOS I • These are sample MyXoS Shell Statements • Specify value for property in GXOS tree<rdf:descriptionabout="gndi://gxosroot/resourcename"><gxos:property rdf:parseType="literal">somevalue</gxos:property> </rdf:description> • Specify profile by linking between GXOS tree elements<rdf:descriptionabout="gndi://gxosroot/sessionname"> <gxos:userprofile rdf:resource="gndi://uri_of_user" gxos:customize="sessionspecificstuff" /></rdf:description> p2pgrideuropejune01

43. RDF Examples in GXOS II • Specify MyXoS copy command for meta-objects<rdf:description rdf:about="gndi://gxosroot/system/bin/cp"system:source="gxosobject1" system:destination="gxosobject2" gxos:execute="true“ /> • Specify alternative locations to find all FSU users<rdf:description aboutEachPrefix="gndi://gxosroot/users/fsu"> <gxos:metaobjectlocation><rdf:alt><rdf:li resource="http://main_fsuweblocation" /><rdf:li resource="http://backup_fsuweblocation" /></rdf:Alt></ gxos:metaobjectlocation></rdf:description> p2pgrideuropejune01

44. Fragment of GXOS Schema p2pgrideuropejune01

45. Interface of XML and Java I • How will we teach computing? • K-4: Internet Access • Middle School: (Simple) XML Schema interfaced to some scripting language • High School: Java as the programming model with Java classes (for external data) generated • Probably don’t want to specify objects twice • Start in Java; generate Schema • Or Start with Schema and generate Java • Seems a natural API of computer code to XML • DOM or SAX XML Parser or • JDBC like Interface of Java to Database (~XML) p2pgrideuropejune01

46. Interface of XML and Java II Database Choose And Convert Software • Suppose we have a quadrillion (1015) XML objects as say produced by a physics accelerator per year (Enterprise GXOS) • Need to combine: • Search Interface to select nodes of XML Tree • Specify URI • JDBC or Google like Interfaces • Castor like Interface to map XML into Java but need to control depth of conversion from XML into XML p2pgrideuropejune01

47. Controlled Conversion XML to Java Node URI Another Node Child1 Child2 … ChildN AnotherObject Model CORBA Java SOAP ….. URI • Control expansion by either • recursively descending tree • Access other nodes not in current sub-tree • Access “real object” in a possibly different object model p2pgrideuropejune01

48. Current GXOS API Architecture • Initially implement “Personal” GXOS – Information Repository small enough that we can afford to read all possibly relevant information into memory and refine this • E.g. Support course data for individual faculty • File.xml  XML Object  Java Object and vice versa • Use Castor to automate XML Schema to Java Object • Primitives Supported Initially • Get a “leaf Object” • Get a Collection (Internal Node) – “handle” and self.xml (the GXOS properties associated with this node) • List Contents of a collection (recursively) • Get Contents of a collection (recursively) p2pgrideuropejune01

49. Lookup: XML  Java Current GXOS API Architecture Client Applications 2. FS JNDI command, lookup(“file:///root/object.xml”) 1. GNDI commands, lookup(“root/object”) alternatively 1a. getUser(“username”) 10. Java object FS JNDI GNDI org.gxos.gndi GXOS Architectural API org.gxos 4. File object 5.File object 9. Java object 1b. lookup(“root/Users/username”) GNDI ObjectFactory GXOS Schema API org.gxos.schema SourceGenerator 3. File find and Return file handle (file object) Castor Java XML classes + Class descriptors for validation 6.FileReader object 8. Java object FS XML objects Unmarshaller GXOS XML Schemas SAX parser Castor 7. XML file access & parsing p2pgrideuropejune01