1 / 14

StoRM Some basics and a comparison with DPM

In this informative talk, Wahid Bhimji from the University of Edinburgh discusses the StoRM storage solution, comparing it with DPM. Key topics include the advantages and issues of moving to StoRM, its lightweight architecture on top of POSIX filesystems, and its growing popularity outside Italy. The presentation also addresses the differences, installation processes, and the maturity of the user community for StoRM. Attendees will gain insights into performance, scalability, and practical considerations for adopting StoRM in storage solutions.

artie
Download Presentation

StoRM Some basics and a comparison with DPM

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. StoRMSome basics and a comparison with DPM Wahid Bhimji University of Edinburgh GridPP Storage Workshop Wahid Bhimji – StoRM

  2. This talk: Considering moving to StoRM ??? • Comparison with DPM • Benefits / Problems • Status Wahid Bhimji – StoRM

  3. WhatsStoRM? • Lightweight SRM server • Works on top of POSIX filesystems • Relies on that filesystem to replicate data etc. • Used widely in Italy and a few place outside Wahid Bhimji – StoRM

  4. DPM - reminder Worker Node rfio Pool Node DISK Pool Node DISK gridftp DPM Head Node Pool Node DISK … User … • “Pool” node (dpm-disk) • Directly mounted disk • RFIO– local transfers • GridFTP– WAN transfers • Head Node (dpm-mysql) • SRMV2 Interface • DPM server: Request tracking (MySQL) • DPM Nameserver (DPNS) (MySQL) Wahid Bhimji – StoRM

  5. Storm Structure Worker Node Disk Servers file:// Disk Servers Disk Servers Disk Servers Disk Servers StoRM: Can be one machine. GridFTPserver(s) GridFTPserver(s) User Storm Frontend Storm Backend Backend: Namespace (by file path) Filesytem Drivers Frontend: SRM Interface Request Tracking (MySQL) Wahid Bhimji – StoRM

  6. Differences to DPM • Namespace derived from filename • Local transfers through file:// POSIX access • Uses ACLs for permissions • Lot less (no) user tools • So no equivalent of dpm-* dpns-* tools • But filesystem operations are just du * rm * ls * • For spacetoken allocations etc. edit file: • /opt/storm/backend/etc/namespace.xml Wahid Bhimji – StoRM

  7. Namespace srm://se2.glite.ecdf.ed.ac.uk:8444/srm/managerv2?=/atlas/atlasdatadisk/bill/bob DPM this would be DPNS path and to get to physical file you would need to use DB • On Storm there is no DB. The physical file is at: SPACE_ROOT/bill/bob SPACE_ROOT is defined for each VO/ spacetoken: Might be just /lustre/atlas/atlasdatadisk Wahid Bhimji – StoRM

  8. Versions • Storm 1.5 released recently • 1.4 no longer “supported” • Currently for SL4 (32bit – prob. works on 64) • SL5 – “after easter” • Installations at QMUL and Edinburgh • both have issues at the moment • probably resolvable. • INFN-T1 are using 1.5 most other sites 1.4 Wahid Bhimji – StoRM

  9. Instalation • Obtain repos http://grid-it.cnaf.infn.it/mrepo/repos/ glite-generic.repo ; lcg-ca.repo; ig.repo; jpackage.repo • “Standard” yum install / YAIM configure : yum install lcg-CA ig_SE_storm_backendig_SE_storm_frontend /opt/glite/yaim/bin/ig_yaim -c -ssiteinfo/site-info.def –nig_SE_storm_backend -nig_SE_storm_frontend • Setup filesystem (Lustre/GPFS etc.; acls needed) • You also need users etc (as for DPM + other nodes) Wahid Bhimji – StoRM

  10. Weighing it up Wahid Bhimji – StoRM

  11. Benefits Easiest current way to make use of Lustre / GPFS or other cluster filesystem: • file:// POSIX access – better local access performance that rfio (afternoon discussion) • Scalability / resilience (INFN-T1 uses Storm) • Simplicity: Navigable directories/ familiar posix commands (but you also have to maintain Lustre / GPFS if you use those) Wahid Bhimji – StoRM

  12. Key reasons not to move • Limited UK experience, smaller user community than DPM /dCache – mostly Italian sites • Some implementation of SRM features lacking (in 1.4 – not clear if fixed in storm 1.5) • Access permissions • Information provider • Space reservation? ;-) : http://www.youtube.com/watch?v=A7uvttu8ct0 Wahid Bhimji – StoRM

  13. StoRM space reservations? ;-) Wahid Bhimji – StoRM

  14. Conclusions • StoRM is not (so) scary. • It has some quirks/limits but so does DPM • You can use the same hardware • SRM performance looks to be similar to DPM • Growing maturity and user community • It is currently the easiest way to use cluster filesystemsto give scalability and the performance of local file:// access • Discuss (after Chris’s talk…) Wahid Bhimji – StoRM

More Related