Light weight Disk Pool Manager status and plans - PowerPoint PPT Presentation

mariam-holder
light weight disk pool manager status and plans n.
Skip this Video
Loading SlideShow in 5 Seconds..
Light weight Disk Pool Manager status and plans PowerPoint Presentation
Download Presentation
Light weight Disk Pool Manager status and plans

play fullscreen
1 / 17
Download Presentation
Light weight Disk Pool Manager status and plans
58 Views
Download Presentation

Light weight Disk Pool Manager status and plans

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Light weight Disk Pool Manager status and plans Jean-Philippe Baud, IT-GD, CERN May 2005

  2. Disk Pool Manager aims • Provide a solution for Tier-2s in LCG-2 • This implies a few tens of Terabytes in 2005 • Focus on manageability • Easy to install • Easy to configure • Low effort for ongoing maintenance • Easy to add/remove resources • Support for multiple physical partitions • On one or more disk server nodes • Support for different space types – volatile and permanent • Support for multiple replicas of hot files within the disk pools CERN IT-GD

  3. Manageability • Few daemons to install • No central configuration files • Disk nodes request to add themselves to the DPM • All states are kept in a DB (easy to restore after a crash) • Easy to remove disks and partitions • Allows simple reconfiguration of the Disk Pools • Administrator can temporarily remove file systems from the DPM if a disk has crashed and is being repaired • DPM automatically configures a file system as “unavailable” when it is not contactable CERN IT-GD

  4. Features • DPM access via different interfaces • Direct Socket interface • SRM v1 • SRM v2 Basic • Also offer a large part of SRM v2 Advanced • Global Space Reservation (next version) • Namespace operations • Permissions • Copy and Remote Get/Put (next version) • Data Access • Gridftp, rfio (ROOTD, XROOTD could be easily added) CERN IT-GD

  5. Security • GSI Authentication and Authorization • Mapping done from Client DN to uid/gid pair • Authorization done in terms of uid/gid • Ownership of files is stored in DPM catalog, while the physical files on disk are owned by the DPM • Permissions implemented on files and directories • Unix (user, group, other) permissions • POSIX ACLs (group and users) • Propose to use SRM as interface to set the permissions in the Storage Elements (require v2.1 minimum with Directory and Permission methods) • VOMS will be integrated • VOMS roles appear as a list of gids CERN IT-GD

  6. Architecture • The Light Weight Disk Pool Manager consists of • The Disk Pool Manager with its configuration and request DB • The Disk Pool Name Server • The SRM servers • The RFIOD and DPM-aware GsiFTP servers • How many machines ? • DPM, DPNS, and SRM can be installed on the same one • RFIOD : on each disk server managed by the DPM • GsiFTP : on each disk server managed by the DPM CERN IT-GD

  7. Example: SRM put processing (1) CERN IT-GD

  8. Example: SRM put processing (2) CERN IT-GD

  9. Example: SRM put processing (3) CERN IT-GD

  10. Example: SRM put processing (4) CERN IT-GD

  11. Example: SRM put processing (5) CERN IT-GD

  12. Example: SRM put processing (6) CERN IT-GD

  13. Example: Pool definition • dpm-addpool --pool Volatile --def_filesize 200M –def_pintime 600 --s_type V • dpm-addfs --pool Volatile --server se001 --fs /data01 • dpm-qryconf • POOL Volatile DEFSIZE 200.00M GC_START_THRESH 0 GC_STOP_THRESH 0 DEFPINTIME 600 PUT_RETENP 86400 FSS_POLICY maxfreespace GC_POLICY lru RS_POLICY fifo GID 0 S_TYPE V • CAPACITY 17.73G FREE 17.73G (100%) • se001 /data01 CAPACITY 17.73G FREE 17.73G (100.0%) CERN IT-GD

  14. Testing • Test suites written by Gilbert Grosdidier (Orsay) and Jiri Kosina (Prague) • Functionality tests (120 sub-tests) • Stress tests (40 suites running in parallel) • The tests were run from very early in the development cycle and were improved as soon as new functionality was available in the servers • The test results were immediately taken into account for the code development CERN IT-GD

  15. Status • DPM will be part of LCG 2.5.0 release but is available from now on for testing CERN IT-GD

  16. Documentation/Packaging/Installation • Reference man pages provided for each method and each server • Installation guide being written • The DPM is available as a tarball or as a set of RPMs • DPM-server-mysql • DPM-name-server-mysql • DPM-srm-server-mysql • DPM-rfio-server • DPM-client • DPM-gridftp-server • Installation scripts (Yaim) are also available CERN IT-GD

  17. Deployment • Replacement of ‘Classic SE’ • Only metadata operations needed (the data does not need to be copied) • Satisfies gLite requirement for SRM interface at Tier-2s CERN IT-GD