gLite Information System

1. gLite Information System Yubiryn Ramírez ULA - Mérida 12th EELA Tutorial Users Lima, 24.09.2007

2. This presentation is based on previous gLite presentations Lima, 12th EELA Tutorial, 24.09.2007

3. Outline • How to discover resource? • Grid Information System • Glue Schema • gLite LCG Information System • Hierarchy • LDAP model • RGMA • Overview • Server Architecture • Consumer-Producer model • RGMA Concepts • References Lima, 12th EELA Tutorial, 24.09.2007

4. How to discover resource? • Once an user is logged into an User Interface (s)he is ready to take advantage of the Grid Power for his/her own application. • But what are the available resources to accomplish his/her tasks? • The answer to this question comes through the interactions with the Information System (IS). Lima, 12th EELA Tutorial, 24.09.2007

5. Grid Information System Overview • What is? • System to collect information on the state of resources • Why? • To discover resources of the grid and their nature • To have useful data in order to who is in charge of managing the workload to do it more efficiently. • To check for health status of resources. • How? • Monitoring state of resources locally and publishing fresh data on the information system. • Adopting a data model that MUST be well known to all components that want to access monitored information Lima, 12th EELA Tutorial, 24.09.2007

6. Designing of Information System • About Measures • Measures SHOULD be sensitive to the aim the users want to achieve. • Measures SHOULD be enough accurate to be considered valid. • Rate of taking measures MUST be adequate to be used. • About the gathering of Information • How and when collected info should be published? • Where should collected info be stored? • How long should this info be maintained in the storage? • Querying the Information System • Where should queries be sent to have a response? • What syntax and protocols have to be adopted to make queries? • What is the adopted data model to describe resources? • Security • Who is allowed to execute queries against the IS and what type of queries is he allowed to do? • Management of user rights and credentials. Lima, 12th EELA Tutorial, 24.09.2007

7. gLite Information System • Basic data model. • Grid Laboratory Uniform Environment (GLUE) Schema. • Two architectures in glite3 • LCG Information System • BDII over Globus MDS. • OpenLDAP interface. • Preserves LCG-2 compatibility. • Relational Grid Monitoring Architecture (R-GMA) • Based on the GMA standard from the Grid Global Forum • Information in SQL relational databases • Web Services. • Future replacement of LCG IS. Lima, 12th EELA Tutorial, 24.09.2007

8. GLUE Schema Lima, 12th EELA Tutorial, 24.09.2007

9. Glue Schema • A schema of objects and attributes describing Grid resources and its relationships. • Originally a EU-DataTAG and US-iVDGL coordinated effort. • Current participants: EGEE, OSG, Globus and NorduGrid. • A way to describe Grid info • Statically and dynamically supplied • Hierarchically represented • Independently of the framework (LDAP, XML, SQL…) • Actual release (1.2) is mapped into • LDAP • XML • Old ClassAd (used by Condor Matchmaking Lima, 12th EELA Tutorial, 24.09.2007

10. Site Element Lima, 12th EELA Tutorial, 24.09.2007

11. Computing Element Lima, 12th EELA Tutorial, 24.09.2007

12. LCG Information System Lima, 12th EELA Tutorial, 24.09.2007

13. LCG Information System Levels • Resource level: Grid Resource Information Server (GRIS) • One GRIS on top of each CE, SE, RB, MyProxy (no WNs). • Sensors and scripts get status of concrete resources statically (e.g. GlueCEUniqueID) or dynamically (e.g. GlueCEStateWaitingJobs) Lima, 12th EELA Tutorial, 24.09.2007

14. LCG Information System Levels • Site level: Grid Index Information Server (GIIS) • Compiles all the information of the different GRISes in a site. • LCG-2 recommends using a BDII instead of a GIIS • Improves robustness and stability. • Called the site BDII Lima, 12th EELA Tutorial, 24.09.2007

15. LCG Information System Levels • Top level: Berkeley DB Information Index (BDII) • Keeps all Grid information about the VOs (generally only one). • Stores information from local BDIIs or GIISes in its database. • Only queries sites that are included in a configuration file. Lima, 12th EELA Tutorial, 24.09.2007

16. LCG Information System Hierarchy Lima, 12th EELA Tutorial, 24.09.2007

17. LDAP Model • Way of collecting info • Pull model (higher level servers periodically query lower level servers) • All servers are based on LDAP • Inherit hierarchical structure (tree-like) • LDAP Data Information Format (LDIF) • Users get info with • Generic applications • ldapsearch (MDS: 2135 and BDII:2170 ports) • Graphical (BDII web; LDAP GUIs) • Always can get information about specific resources (maybe more up-to-date) by querying directly the site BDIIs, GIISes or GRISes. • Querying VO info with lcg-infosites or lcg-info Lima, 12th EELA Tutorial, 24.09.2007

18. LDAP Browser show BDII information Lima, 12th EELA Tutorial, 24.09.2007

19. R-GMA Lima, 12th EELA Tutorial, 24.09.2007

20. Introduction to R-GMA • Relational Grid Monitoring Architecture (R-GMA) • Developed as part of the EuropeanDataGrid Project (EDG) • Now as part of the EGEE project. • Based the Grid Monitoring Architecture (GMA) standart from the Global Grid Forum (GGF). Lima, 12th EELA Tutorial, 24.09.2007

21. R-GMA Overview • Information in SQL relational databases (a DB per VO) • Query syntax is a SQL subset • Simple consumer-producer model • Web Services oriented • CLI and Web user interface • Allows self-logging applications • Future replacement of LCG IS Lima, 12th EELA Tutorial, 24.09.2007

22. R-GMA Overview • R-GMA offers a global view of the VO information • In one large relational DB: virtual database. • Registry stores localization tuples (database rows) published by producers: • Standard Tables: CE state in GLUE Schema (by R-GMA-GIN) • Applications specific tables (e.g. self-logging with Log4j) • Access by SQL queries through a WS interface. • Always updated • Consumers get information directly from the resource • Can be cached by a secondary producer from single o multiple producers to improve access. Lima, 12th EELA Tutorial, 24.09.2007

23. R-GMA Server Architecture Lima, 12th EELA Tutorial, 24.09.2007

24. R-GMA Server Description • R-GMA Servers • Server (one or several) • Schema Server (There is one common global schema server for the grid) • Registry Server (one or several) • Browser (shows grid information via web browser) Lima, 12th EELA Tutorial, 24.09.2007

25. R-GMA Server Description R-GMA Clients • Generic Client (generic set of APIs for different languages and CLIs) • Servicetool (used by all services to publish its existence and status) • Site Publisher • Client to publish the existence of a site. • Each site must have one R-GMA site publisher. • GadgetIN (GIN) • Client to extract information from MDS and to republish it to R-GMA. • used by the CE. • Data archiver (flexible archiver) • Client to make the data that is coming from the R-GMA site-publisher, servicetool and GIN constantly available. • By default the GLUE tables and service tables are archived, however this can be configured. Lima, 12th EELA Tutorial, 24.09.2007

26. R-GMA Server per Node Type • The gLite R-GMA Server should deployed the first as part of a gLite grid, because all services require it to publish service information. Lima, 12th EELA Tutorial, 24.09.2007

27. CONSUMER PRODUCER REGISTRY RGMA Producer – Consumer model • The Producer stores its location (URL) in the Registry. Store location • The Consumer looks up producer URLs in the Registry. • The Consumer contacts the Producer to get all the data or the Consumer can listen to the Producer for new data. Transfer Data Lookup location Lima, 12th EELA Tutorial, 24.09.2007

28. RGMA Multiple Producers • The Consumer will get all the URLs that could satisfy the query. • The Consumer will connect to all the Producers. • Producers that can satisfy the query will send the tuples to the Consumer. • The Consumer will merge these tuples to form one result set. Registry Producer 1 Producer 2 Consumer Lima, 12th EELA Tutorial, 24.09.2007

29. Select * from CPULoad Lima, 12th EELA Tutorial, 24.09.2007

30. Joins SELECT Service.URI Service.emailContact FROM Service S, ServiceStatus SS WHERE (S.URI= SS.URI and SS.up=‘n’) Lima, 12th EELA Tutorial, 24.09.2007

31. R-GMA Concepts • Producers • Primary producer The producer creates and stores locally the informatiion • Secondary producer The producer stores locally the information coming from primary producers • On Demand The producer returns the information only under the request of the consumer • Consumer • ContinuousAs soon as new data becomes available it is broadcast to all interested parties • LatestCorresponds to the intuitive idea of “current information” • History • Returns time sequenced data Lima, 12th EELA Tutorial, 24.09.2007

32. References • LCG-2 User Guide • https://edms.cern.ch/file/454439//LCG-2-UserGuide.html • R-GMA overview page. • http://www.r-gma.org/ • R-GMA in EGEE • http://hepunx.rl.ac.uk/egee/jra1-uk/ • R-GMA Documenation • http://hepunx.rl.ac.uk/egee/jra1-uk/glite-r1/ • GLUE Schema • http://infnforge.cnaf.infn.it/glueinfomodel/ Lima, 12th EELA Tutorial, 24.09.2007

33. Questions … Lima, 12th EELA Tutorial, 24.09.2007