1 / 15

DPM Basics and its status and plans

DPM Basics and its status and plans. Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010. This talk: . Not covering all the basics of DPM many already know those. S ee gridpp wiki: http://www.gridpp.ac.uk/wiki/Disk_Pool_Manager or Sam’s talks in July:

lan
Download Presentation

DPM Basics and its status and plans

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. DPMBasics and its status and plans Wahid Bhimji University of Edinburgh GridPP Storage Workshop – Apr 2010 Wahid Bhimji – DPM

  2. This talk: • Not covering all the basics of DPM • many already know those. See gridpp wiki: http://www.gridpp.ac.uk/wiki/Disk_Pool_Manager or Sam’s talks in July: http://hepwww.rl.ac.uk/sysman/June2009/agenda.html • Will cover: • An introduction of terms. • Listsomecommonly used tools for reference • Details of recent developments and plans for discussion Wahid Bhimji – DPM

  3. A “standard” DPM setup Pool Node Worker Node DISK rfio Pool Node DISK gridftp DPM Head Node Pool Node DISK … User … • “Pool” node (dpm-disk) • Directly mounted disk • RFIO– local transfers • GridFTP– WAN transfers • Head Node (dpm-mysql) • SRMV2 Interface • DPM server: Request tracking (MySQL) • DPM Nameserver (DPNS) (MySQL) Wahid Bhimji – DPM

  4. Available tools • Some built-in and some in the gridpp toolkit: http://www.gridpp.ac.uk/wiki/DPM-admin-tools Latest version 2.6.3 • Let Sam or I know if you need another tool • For help ask the GRIDPP-STORAGE jiscmail list • or dpm-users-forum@cern for generic dpm questions Wahid Bhimji – DPM

  5. Common activities • Install / upgrade: See GridPP Wiki • Look at space usage, list files • Move data around • Allocate or modify space: • “pools” given to VOs (or collection of VOs) • “Spacetokens” to groups within VOs Wahid Bhimji – DPM

  6. Finding out whats on your storage • dpns-ls: List files in namespace • dpm-disk-to-dpns– translate filepath to dpns • dpm-list-disk - list all files on a filesystem Overall picture • dpm-listspace: Includes usage in spacetokens • dpm-qryconf: Includes usage in filesystem • dpns-du: For space usage within a namespace path. Wahid Bhimji – DPM

  7. Moving files • dpm-delreplica or rfrm : remove from disk and namespace (but not LFC / remote catalogues) • dpm-replicate or rfcp : copy single file • dpm-drain: move a whole filesystem -Need to wrap in an “until” loop to persist if it crashes. • dpm-sql-spacetoken-replicate-hotfiles– (new(ish) in toolkit) – replicate all files within a spacetoken Wahid Bhimji – DPM

  8. Allocating Space ALL ATLAS VO space should be in SPACETOKENS Create: dpm-reservespace Update dpm-updatespace- do not use dpm-reserve again Delete: dpm-releasespace Moving a filesystem between pools: dpm-rmfs --server pool2 --fs /gridstorage018 dpm-addfs --poolnamecms --server pool2--fs /gridstorage018 Wahid Bhimji – DPM

  9. DPM Status Wahid Bhimji – DPM

  10. Support • Last year central DPM support from CERN was at the level of 0.5 FTE. • Now improved to ~2 FTE: • Jean-Philippe Baud has returned • Additional developer : Ricardo Rocha Wahid Bhimji – DPM

  11. Versions • Latest version – still 1.7.2(-5) • 1.7.3 (in certification since Oct.) Provides: • Checksum treatment • Fix for unstable SRM server (currently patched binary) • Future Program of work https://twiki.cern.ch/twiki/bin/view/LCG/DMProgramOfWork Wahid Bhimji – DPM

  12. Future Plans - Highlights Wahid Bhimji – DPM

  13. Long(er) term plans - Confirmed • Quotas. • Accounting: Space uses, files:read, written, replicated - bytes • Replace HTTPG by HTTPS for SRM servers • NFS 4.1 interface • VO admin role • Glue2 support “Not Scheduled” • Add GUIDs to file entries to be able to use DPM as local LFC • Automatic replication of hot files. attributes, not I/O. • Integration with Argus (banning of users and groups). • Better file system selection algorithm.. • Support for distributed file systems like Lustre/Hadoop Wahid Bhimji – DPM

  14. Unconfirmed • Support for striping • Support for BitTorrent • Multi-vo support for xrootd • WEB admin interface • Gateway hosts to allow sites having very few IP addresses visible externally to use DPM • Provision of native python interfaces • Support for python 2.6 Wahid Bhimji – DPM

  15. Conclusions / Discussion • DPM has many tools – use not always obvious • Significant support in UK – use us • Now good support at CERN too • And a plan of action • We can influence this • We can also help develop features of interest • E.g asked to help with Nagiousplugins • What are the priorities? Wahid Bhimji – DPM

More Related