1 / 49

Grid Calculation Framework - Lyon User Training, October 19th, 2010

This training session in Lyon focuses on grid calculation frameworks, examples, interoperability, and standards related to grid architectures. It includes discussions on the SAGA specification and JSAGA implementation, as well as applications and perspectives. The session also covers security, data management, resource management, and execution management in grid environments.

mbyrnes
Download Presentation

Grid Calculation Framework - Lyon User Training, October 19th, 2010

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grilles de Calcul - Formation utilisateurs Lyon, 19 octobre 2010 Sylvain Reynaud

  2. Plan • Contexte • exemples de grilles • interopérabilité : motivations • Standards de grille • Open Grid Forum • la spécification SAGA • JSAGA : une implémentation de SAGA • Applications utilisant JSAGA • Perspectives JSAGA

  3. GOS NAREGI Quelques grilles dans le monde middleware infrastructures JSAGA

  4. Quelques grilles dans le monde middleware GOS NAREGI infrastructures JSAGA

  5. N Quelques grilles en Europe middleware GOS NAREGI infrastructures JSAGA

  6. Quelques grilles en Europe middleware GOS NAREGI infrastructures JSAGA

  7. N Quelques grilles en France JSAGA

  8. cluster Interopérabilité : motivations j'ai besoin de beaucoup de ressources batch SSH JSAGA

  9. cluster Interopérabilité : motivations certains de mes jobs ont besoin de logiciels ou services particuliers batch SSH JSAGA

  10. cluster Interopérabilité : motivations certains de mes jobs doivent tourner sur un supercalculateur batch SSH JSAGA

  11. cluster Interopérabilité : motivations je n'utilise la grille que pour absorber des pics de charge batch SSH JSAGA

  12. cluster Interopérabilité : motivations je veux éviter l'overhead de la grille pour certains jobs batch SSH JSAGA

  13. cluster Interopérabilité : motivations je débugue en local avant de soumettre sur la grille fork batch SSH JSAGA

  14. Standards de grille • Besoins et concepts communs • Une communauté d'industriels et de chercheurs définit des standards • pour l'architecture globale d'une grille (Open Grid Service Architecture) • pour chaque composant d'un middleware de grille JSAGA

  15. Sécurité grid-proxy-init VOMS CE, SE… Authentification Autorisation (décision partagée) Interopérer avec les infrastructures de sécurité locales (sans les compromettre) Délégation Négociation Isolation des utilisateurs Traçabilité, détection d’intrusion LCAS / LCMAPS JSAGA Source: The Open Grid Services Architecture, Version 1.5

  16. Servicediscovery Loadbalancing Problemdetermination Executionmanagement Accounting Resourcereservation Applicationmonitoring Systèmes d'information Architectures et interfaces dépendent de… type d'information (fréquence de rafraîchissement, durée de vie des infos) type d'utilisation (critères de QoS différents) BDII InformationServices Registry Asynchronous notification Consumers Producers Retrieval • Reliable • Secure • Efficient Logger APEL LB JSAGA Source: Hiro Kishimoto

  17. SRM Data Resources Managed Storage Serviceinterface Resourceinterface Gestion des données LFC, AMGA FTS Non-OGSA client APIs & other services Gestion de stockage Accès aux données Gestion de réplicats Catalogues de méta- -données Transfert de données Gestion de cache Transfer Lookup Transfer Registries Sink/ Source Description Data Management Other Data Services Storage Storage Management Access Access Description Sink/ Source Data Resources Transfer Protocols GridFTP JSAGA Source: The Open Grid Services Architecture, Version 1.5

  18. ? ? ? Information Service WMS Gestion de l'exécution Description du job Sélection de la ressource Soumission, suivi et contrôle du job Déploiement JDL Application Contents Service Config Desc, Deployment & Lifecycle Mgmt JSDL LCG-CE, CREAM-CE JSDL Job Manager Basic Execution Service resource user BDII Resource Selection Service JSAGA Source: Keisuke Fukui (ACS-WG)

  19. Interface utilisateur gLite-UI VOMS, MyProxy user WMS, LCG-CE, CREAM-CE SRM, gsiftp LFC, AMGA BDII JSAGA Source: S Jha, T Kielmann, A Merzky

  20. SAGA est une interface standard spécification de l' orientée objet simple pour 80% des besoins, 20% des fonctionnalités suffisent intégrée "look & feel" commun à tous les packages de l'API high-level les détails du fonctionnement du middleware sont cachés indépendant du middleware même interface pour tous les middlewares indépendant du langage Java C++ python (wrappers) stable spécification SAGA 1.0 (15 janvier 2008) extensible Service Discovery, Message… Simple API for Grid Applications JSAGA

  21. Simple API for Grid Applications • Exemple de code en Java (data management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("gsiftp://cclcgseli01.in2p3.fr/tmp/"); // instanciation de l'objet SAGA à l'aide d'une fabrique NSDirectory dir = NSFactory.createNSDirectory(session, url); // utilisation de l'objet SAGA List<URL> result = dir.list(); for (URL r : result) System.out.println(r); liste le contenu du répertoire gsiftp://… JSAGA

  22. Simple API for Grid Applications • Exemple de code en Java (execution management) Session session = SessionFactory.createSession(true); URL url = URLFactory.createURL("wms://lapp-wms02.in2p3.fr/…"); // instanciation de l'objet SAGA à l'aide d'une fabrique JobService svc = JobFactory.createJobService(session, url); // utilisation de l'objet SAGA Job job = svc.runJob("/bin/date --utc"); job.waitFor(); System.out.println(job.getState()); soumet un job et attend la fin de son exécution JSAGA

  23. Simple API for Grid Applications JSAGA

  24. monitor protocol control security Simple API for Grid Applications Execution Data Security JSAGA

  25. The JSAGA implementation • a Java implementation of the SAGA specification • focuses on • uniform usage of middleware…. (beyond what they natively support) • ease of extension………………. • efficiency and scalability………. • control of the API behavior........ (configure binding rather than trying) • operating-system independency (tested on ) • Under LGPL license (thanks to the design of adaptor interfaces) JSAGA

  26. Uniform usage of existing grids JSAGA

  27. out.txt out.txt JSAGA GridFTP Example – uniform usage SAGA job SAGA job user applications WMS gsiftp Gatekeep. Globus VOMS JSAGA

  28. DGrid proxy adapt + translate submit job receive status transfer to any protocol EGEE proxy + attributes translate submit job poll status delegate transfer out.txt out.txt EGEE D-Grid JSAGA Example – what happens behind SAGA job user applications modified job wrapper script WMS gsiftp Gatekeep. Globus VOMS JDL RSL VOMS WMS GridFTP Globus GK Unicore LCG-CE CREAM-CE JSAGA

  29. JSAGA interface Layered software architecture implementation JSAGA

  30. Used by end users web portals, GUI, CLI… Used by application developers a single uniform interface object oriented, high-level Implemented by adaptors each "way" to implement a given feature has 1 interface service oriented, low-level Implemented by middleware each technology has its own interfaces SAGA Java binding core engine JSAGA adaptors interfaces JSAGA adaptors JSAGA interface Layered software architecture implementation user applications middleware APIs JSAGA

  31. JSAGA adaptor interfaces adaptor adaptor GridFTP Globus GK Layered software architecture • Used by end users • web portals, GUI, CLI… • Used by application developers • a single uniform interface • object oriented, high-level • Implemented by adaptors • each "way" to implement a given feature has 1 interface • service oriented, low-level • Implemented by middleware • each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine upload data get output get status JSAGA

  32. adaptor adaptor Layered software architecture • Used by end users • web portals, GUI, CLI… • Used by application developers • a single uniform interface • object oriented, high-level • Implemented by adaptors • each "way" to implement a given feature has 1 interface • service oriented, low-level • Implemented by middleware • each technology has its own interfaces user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces GridFTP Globus GK upload data get output get status JSAGA

  33. adaptor adaptor Layered software architecture • Middleware API can be used more efficiently • engine selects best interface for each user request • Developing new adaptors is easier • just wrap functionalities supported by middleware API user applications transfer output listen status changes JSAGA core engine JSAGA adaptor interfaces GridFTP Globus GK upload data get output get status JSAGA

  34. As many interfaces as ways to implement each functionality Example: ways to monitor jobs API mode poll job status listen to notifications about job status changes API granularity individual jobs list of jobs jobs filter (e.g. by user, by date, by tag…) done construction planned SAGA adaptors interfaces JSAGA Design of adaptors interfaces getState waitFor query status for individual job listen status for individual job query status for filtered jobs Job monitoring gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local JSAGA

  35. Optional interfaces for optimization Example: ways to copy a file data read/write methods either stream methods or get/put methods data copy e.g. enable third-party transfer delegated transfer done construction planned SAGA adaptors interfaces JSAGA Design of adaptors interfaces copy data reader data writer data copy Physical rbyteio gsiftp srm srb irods http https sftp ftp file zip cache JSAGA

  36. Minimal dependencies on external libraries external tools e.g. no gLite-UI operating system tested on done construction planned gatekeeper gLite-WMS wsgram unicore6 cream naregi ssh local InMemCred Globus G. Legacy G. RFC820 MyProxy VOMS JKS SSH Login / pwd X509 Supported technologies Grid techno. Others Execution Grid techno. Others Data Logical Physical rns lfn srb irods catalog rbyteio gsiftp srm srb irods http https sftp ftp file zip cache Grid techno. Others Security JSAGA

  37. Adaptors validated by a middleware-independent SAGA test suite maven 'archetype' to generate skeleton of new adaptor project Automated build process download and install build tools external libraries generate source code execute test-suites unitary tests integration tests generate project web site documentation reports generate installer GUI extract library dependencies from maven project description Software quality # SAGA protocols test-suite configuration gsiftp.base=gsiftp://ccrugceli01.in2p3.fr/tmp/ gsiftp.base2=gsiftp://agena.c-s.fr/grid/tmp/ gsiftp.context=OpenPlast_proxy https.base=http://grid.in2p3.fr/html/Private/ https.context=Web_X509 file.base=file:///c:/tmp/ file.base2=file:///c:/ JSAGA

  38. Installer GUI JSAGA

  39. LGPL license for the core engine and for most adaptors Optional licenses for adaptors having external dependencies, which license is not compatible with LGPL then, end-user must… either accept the terms of the license agreement, or go back to previous screen and uncheck the adaptor(s) Licenses JSAGA

  40. Essayer tous les contextes attractif pour les débutants problème d'efficacité gaspillage de temps gaspillage de ressources (risque d'échec à la fin du job) problème de fiabilité risque de créer les fichiers avec le mauvais propriétaire risque de bloquer un compte (si trop de tentatives échouent) Associer les URL aux contextes contrôle du comportement nécessite qq. connaissances sur les infrastructures utilisées la configuration de JSAGA Le nouveau moteur de configuration permet configuration dynamique supporte 1 configuration/session détecte les conflits à la config. Simplicité versus contrôle JSAGA

  41. X509 N Démonstration Contextes de sécurité utilisés Password: **** VO Rhône-Alpes VO dteam CA allemande CA perso SSH iRods SRB https JSAGA

  42. Coexistence of SAGA implementations • several implementations • different languages (sometimes) • different engines • different context selection mechanisms (trying / configuring) • different adaptor interfaces (high-level / low-level interfaces) • … • different supported technologies (i.e. adaptors) • how to use all these technologies from a grid application ? Application Application S A G A C++ Bindings SAGA Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA JSAGA

  43. C Python Jython Coexistence of SAGA implementations Application Application S A G A Python Bindings for SAGA (PySAGA) CppySAGA JPySAGA JySAGA C++ Bindings SAGA Java Bindings for SAGA SAGA-C++ JSAGA Java SAGA JSAGA

  44. DIRAC (Distributed Infrastructure with Remote Agent Control) the pilot jobs system of LHCb experiment Pandora Gateway the grid platform of the company a Grid/Cloud Computing Business Services Provider JUX (Java Universal eXplorer) a multi-protocols extensible file explorer Example applications JSAGA

  45. Elis@ web portal for industrial and academic production grids JJS (Java Job Submission) tools for submitting jobs to EGI resource selection based on QoS observed while submitting jobs SimExplorer software for simulation experiment management includes a workflow engine that uses JSAGA to run simulations on grids jCAE(Java CAE) Computer Aided Engineering with support for distributed computing facilities / Example applications JSAGA

  46. data management jsaga-cat jsaga-cp jsaga-ls jsaga-mkdir jsaga-mv jsaga-rm jsaga-rmdir jsaga-stat jsaga-test jsaga-logical Example applications • JSAGA provides command line interfaces for… • security • jsaga-context-init • jsaga-context-info • jsaga-context-destroy • execution management • jsaga-job-run • jsaga-job-status • jsaga-job-cancel • jsaga-job-output • Also useful as code examples to start with SAGA JSAGA

  47. Develop new plug-ins NorduGrid middleware ARC plug-in in progress… LRMS (DRMAA-based) Grid Engine LRMS (CLI-based/ssh) torque Grid Engine Develop API extensions Grid-RPC (from SAGA core) Service Discovery API (a SAGA extension) with plug-ins for technologies BDII with plug-ins for schemas GLUE CIM Perspectives JSAGA

  48. Job submission principle describe user's job submit it to the grid middleware select the execution host OGF standards BES, DRMAA, SAGA examples Remote Procedure Call (RPC) principle deploy user's service on grid call a procedure middleware select the service instance OGF standards Grid-RPC, SAGA examples Perspectives job Grid-RPC JSAGA

  49. Questions ? JSAGA

More Related