1 / 16

WP5 Storage Element

The WP5 mission aimed to streamline data access across mass storage systems by providing a common API, following the SRM interface. Users are familiar with Site File Names (SFN) and Physical File Names (PFN) for file retrieval. Key developments include client-server interactions for data requests, the management of disk cache using services such as GridFTP and RFIO, and newer functionalities in SRM. The talk outlines implementations from various sites, ongoing migrations, and future directions towards improving scalability and service efficiency within the storage framework.

eyad
Download Presentation

WP5 Storage Element

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WP5 Storage Element DataGrid Project Conference Heidelberg, 26 Sep-01 Oct 2003

  2. Intro • WP5 mission was to provide a common API to Mass Storage • People as why it is so complicated • … because we provided a common API to Mass Storage • And because we followed the SRM interface

  3. More about SRM • Users know Site File Names (SFN) or Physical File Names (PFN) • lxshare0408.cern.ch/bongo/mumble SE cache file request id Disk cache MSS file

  4. More about SRM • Client queries the status of a request • Better that client polls than server callbacks • Server (ideally) able to give time estimate SE status? “not ready, try later” Disk cache MSS file

  5. More about SRM • When request is ready, client gets a Transfer URL (TURL) • gsiftp://lxshare0408.cern.ch/flatfiles/01/data/16bd30e2a899b7321baf00146acbe953 SE status? ready: TURL Disk cache MSS file file

  6. More about SRM • Client accesses the file in the SE’s disk cache using (usually) non-SE tools SE Disk cache MSS globus- url-copy file file file

  7. More about SRM • Finally, client informs SE that data transfer is done • This is required for cache management etc SE done ok Disk cache MSS file file

  8. EDG-SE in TB2.0 • SE Control System delivered as webservice • Provides common (blocking) interface to all SEs • Disk and several MSS • Publishes GLUE Storage Schema to Information Service (RGMA) • MSS supported • Castor, HPSS, ADS, disk • Protocols supported • Gridftp, file, rfio • Each VO has its own name space • WP9 use case

  9. Sites • Tested in application testbed • Barcelona(Castor), NIKHEF, Imperial, RAL-Tier1(ADS), RAL-PP, Karlsruhe, CC-IN2P3 Lyon • Tested in 2.0 dev testbed but not yet integrated in app testbed • CERN (Castor), CC-IN2P3 (HPSS) • Under development but not in TB2.1 • ESRIN – AMS • SARA – SGI • INSA - Dicom

  10. Protocols • SE returns Transfer URL (TURL) for requested protocol • GridFTP(gsiftp) can be used for data transfer to remote SE • File transferred using globus-url-copy • FILE protocol provides access to disk on ‘local’ SE from WN when WSE filesystems nfs-mounted on WN • TURL= file://pathname • rfiod runs on all Ses and clients on WNs • TURL= rfio://host/path (not hostname:/pathname)

  11. WP9 Migration • Other applications didn’t want to migrate data from TB1 • WP9 had 11,000 files dstributed round TB1.4 • Use as a migration test case without physically moving files • WP2 tool migrates TB1 Replica Catalogue to RLS • And checks existence of files. • WP5 tool run by sysadmins at each site to register files in TB2.0 SE • Plan extended due to sysadmins wanting to move files from TB1.4 locations • Migration still running last night

  12. EDG-SE in TB2.1 • Bug fixes • Delete • More efficient ls to speed up edg-rm registration • Secure mode SE • User identified by DN • VO taken from gridmapfile • cf insecure mode where all files owned by a site-defined user (default John Gordon)

  13. Ready for demo Developments completed but not released in TB2.1 • ACL • Using GACL • SRM1.1 for GFAL • Implemented subset of SRMv1.1 used by GFAL spec • Grid File Access Library being developed by CERN for LCG • Implemented as adaptor class on TB2 SE

  14. Will be ready for demo • Asynchronous (non-blocking) SE • Full SRM v1.1 • As separate direct webservice • VOMS support • Groups, roles, etc • Already supported by GACL • Use edg-authorisation-manager to parse VOMS credentials • Disk Cache Management

  15. Future Directions – post EDG • Sell to LCG • For SRM and VOMS access to MSS • Guaranteed reservations • SRM2 recommendations: volatile, durable, permanent files and space • Full SRM version 2.1 • Scalability • Scalability will be improved by making a single SE distributed

  16. Outro • I am retiring as WP Manager • Being replaced by Jens Jensen • So long and thanks for all the fish!

More Related