1 / 14

Grid Interoperation on Data Movement between NAREGI and EGEE gLite

Grid Interoperation on Data Movement between NAREGI and EGEE gLite. Hideo MATSUDA 1,2 , Yoshiyuki KIDO 3,2 , Kentaro WAKATSUKI 4 1 NAREGI, 2 Osaka University, 3 Mitsui Knowledge Industry Co.,Ltd. 4 Hitachi Software Engineering Co.,Ltd. GIN (Grid Interoperation Now).

aram
Download Presentation

Grid Interoperation on Data Movement between NAREGI and EGEE gLite

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Grid Interoperation on Data Movement between NAREGI and EGEE gLite Hideo MATSUDA1,2, Yoshiyuki KIDO3,2, Kentaro WAKATSUKI 4 1 NAREGI, 2 Osaka University, 3 Mitsui Knowledge Industry Co.,Ltd. 4 Hitachi Software Engineering Co.,Ltd.

  2. GIN (Grid Interoperation Now) • An activity of OGF for interoperation among production grids • Major grid projects are participating • EGEE, NAREGI, UK National Grid Service, NorduGrid, OSG, PRAGMA, TeraGrid, ... • Trying to identify islands of interoperation between production grids and grow those islands • Areas • GIN-auth: Authorization and Identity Management • GIN-data: Data Management and Movement • GIN-jobs: Job Description and Submission • GIN-info: Information Services and Schema • GIN-ops: Operations Experience of Pilot Test Applications To change: View -> Header and Footer

  3. NAREGI GIN Activities • Developing an interoperation island with EGEE • GIN-jobs, GIN-auth, GIN-info, and GIN-data To change: View -> Header and Footer

  4. gLite-WMS gLite-BDII GIN-BDII NAREGI-IS gLite-UI NAREGI Portal gliteCE NAREGI-SS NAREGI-GAHP NAREGI-SC NAREGI Client Lib Interop-SC EGEE user NAREGI user lcgCE Computing Resource WS GRAM NAREGI GridVM PreWS-GRAM GIN-jobs: NAREGI-EGEE Architecture & SC06 Demo • Architecture • SC06 Demo • NAREGI  EGEE: using NAREGI Workflow • EGEE  NAREGI: using gLite WMS commands To change: View -> Header and Footer

  5. NAREGI PMA GIN-auth: Authentication • IGTF is framework of International Grid Trust Federation. • IGTF consists of APGridPMA, EUGridPMA and TAGPMA. • NAREGI CA joined the APGrid PMA. • NAREGI CA has been approved as a production-level CA by APGridPMA. IGTF (International Grid Trust Federation) • GSI compliant with x.509 proxy certificates for authentication. • It has become available to use grid computing easily on the worldwide Internet by IGTF. EUGridPMA TAGPMA APGridPMA To change: View -> Header and Footer

  6. GIN-info: Architecture All of grid information can be retrieved by each of grid in its fashion WRT resource description schema, data format, query language, client API, … Each information service in grid acts as an information provider for the other and translator embedded in the provider performs conversion between different schemas. NAREGI OGSA -DAI Aggregator Glue v1.1 “Site on a map” TeraGrid/ MDS4 xmlCIM RDB OS Glue v1.2 LRPS Processor CIM v2.12 /w ext. Storage ARC GIN-BDII ARC -BDII LDIF JobQueue Service Cell Domain connecting with BDII ● 　● CIM Providers with Glue=>NRG translator Generic Information Provider LDIF providers with X  Glue translators : OSG Pragma NDGF EGEE NAREGI TeraGrid To change: View -> Header and Footer

  7. GIN-data • Data Management and Movement • Agreements: • Grid FTP is the lowest common denominator for file transfer • SRM and SRB islands for data management are being established To change: View -> Header and Footer

  8. NAREGI Software and Data Grid Application requirement definition Work-flow Input files CA b2: Data import c: Edit DataGrid WFT 8: Visualization Portal 1: Submission a: Sign-on Information Service 2: Resource discovery SuperScheduler 3: Negotiation 5: IMPI starts Agreement 9: Accounting Co-Allocation GridVM GridVM GridVM 6: MPI job starts 4: Reservation LocalScheduler LocalScheduler LocalScheduler Network monitor 2: Monitoring 7: MPI init. FMO Job IMPIServer RISMJob DataGrid GridMPI Site ρ Site α Site μ Gfarm File System To change: View -> Header and Footer

  9. Gfarm File System (1) • Developed by AIST, Japan. • Commodity-based distributed file system that federates local disks of compute nodes • It can be shared among all cluster nodes and clients • Just mount it as if it were high-performance NFS • It provides scalable I/O performance wrt the number of parallel processes and users • It supports fault tolerance and avoids access concentration by automatic replica selection To change: View -> Header and Footer

  10. Gfarm File System (2) • Files can be shared among all nodes and clients • Physically, it may be replicated and stored on any file system node • Applications can access it regardless of its location • File system nodes can be distributed Client PC /gfarm Gfarm file system metadata File A File A Note PC File B File C File C File A File B File B … EU File C Japan To change: View -> Header and Footer

  11. NAREGI NAREGI Client SRM Client Gfarm API NAREGI Metadata Server DPM (SRM Server) GIN-data: Architecture • NAREGI and EGEE gLite clients can access to both data resources (e.g., bi-directional file copy) using SRM interface. • GridFTP is used as its underlying file transfer protocol. • File catalog (metadata) exchange is planned. EGEE gLite Client SRM Client LFC (Metadata Server) GridFTP Server Gfarm Server Storage Storage To change: View -> Header and Footer

  12. GIN-data: File Transfer with GridFTP-DSI • It is not easy to bridge between different file access protocols SRM (gLite) and Gfarm. • As the 1st step, bi-directional file transfer between gLite and Gfarm using GridFTP. • Problem: GridFTP authentication (delegation) cannot be directly passed to Gfarm file server. • GridFTP-DSI (Data Storage Interface) has been used for integrating Gfarm API into GridFTP. GridFTP client Proxy cert GridFTP server export DSI for Gfarm Proxy cert Gfarm metadata sever Gfarm API Gfarm client lib Gfarm file server GridFTP server for Gfarm access Gfarm file system To change: View -> Header and Footer

  13. GIN-data: SC06 Demo • SRM copy (srmcp) command was ported in NAREGI. • Bi-directional file transfer can be performed using GridFTP with DSI for Gfarm. NAREGI (pbg1052.naregi.org) GridFTP server for Gfarm access EGEE (lxdpm01.cern.ch) SRM client srmcp gsiftp://pbg1052 srm://lxdpm01 srmcp srm://lxdpm01 gsiftp://pbg1052 Gfarm DSI SRM (DPM) Server Gfarm Server To change: View -> Header and Footer

  14. Summary • NAREGI developed EGEE-NAREGI island as an activity of GIN • Bilateral information exchange • Bilateral job submission • Bilateral file exchange • Interoperable security properties • Next steps • Improve interoperation interfaces and functions WS-GRAM, BES, JSDL, … • Grow the island with other EGEE partners • KEK will use NAREGI-EGEE interoperation environment for their high energy physics calculations To change: View -> Header and Footer

More Related