Castor gridftp
This presentation is the property of its rightful owner.
Sponsored Links
1 / 15

CASTOR / GridFTP PowerPoint PPT Presentation


  • 99 Views
  • Uploaded on
  • Presentation posted in: General

CASTOR / GridFTP. Emil Knezo PPARC-LCG-Fellow CERN IT-ADC GridPP 7 th Collaboration Meeting, Oxford UK July 1st 2003. Outline of this talk. Introduction to CASTOR HSM CASTOR/GridFTP approach GridFTP problems CASTOR/GridFTP test service Configuration issues Usage examples

Download Presentation

CASTOR / GridFTP

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Castor gridftp

CASTOR / GridFTP

Emil Knezo

PPARC-LCG-Fellow

CERN IT-ADC

GridPP 7th Collaboration Meeting, Oxford UK

July 1st 2003


Outline of this talk

Outline of this talk

  • Introduction to CASTOR HSM

  • CASTOR/GridFTP approach

  • GridFTP problems

  • CASTOR/GridFTP test service

  • Configuration issues

  • Usage examples

  • Plan for CASTOR/GridFTP service

CASTOR & GridFTP / Emil Knezo CERN


Castor

CASTOR

  • CASTOR Mass Storage System evolved from SHIFT (tape management system of 90’s)

  • CASTOR is HSM

  • Today @ CERN: 2066.37 TB of data of 10.51 M files stored in CASTOR

  • CASTOR provides to users:

    • Name space

      • File names are in the form:

        /castor/domain_name/experiment_name/…

        for example: /castor/cern.ch/cms/

        /castor/domain_name/user/…

        for example: /castor/cern.ch/user/k/knezo

    • POSIX compliant I/O: RFIO

      + 64-bits support, streaming mode;

      - security

CASTOR & GridFTP / Emil Knezo CERN


Castor current layout

NAME

server

RFIO

Client

VDQM

server

NAME

server

VDQM

server

STAGER

TPDAE

MON

(PVR)

RTCOPY

CLIENT

RTCPD

RTCPD

(TAPE

MOVER)

RFIOD

(DISK

MOVER)

MSGD

VOLUME

manager

DISK POOL

CASTOR current layout

CASTOR & GridFTP / Emil Knezo CERN


Gridftp for castor

GridFTP for CASTOR

  • Motivation for GridFTP interface to CASTOR

    • LCG

      • Data-movement protocol to couple different HSM systems of Tier-1 centers

      • Used by Replica Management System

    • Experiments

      • Offer experiments a secure alternative to rfio and FTP

      • Support CMS world-wide production starting in July

        Mid-July 2003:1TB per day to CASTOR from 12 regional centers

        February 2004: several TB per day from/to CASTOR

  • Approach for GridFTP interface to CASTOR

    • Modification of external GridFTP server to act as rfio-client to CASTOR

      • Solution already proven for FTP servers

      • Not enough man-power do develop and maintain our own server

      • Development time restriction

CASTOR & GridFTP / Emil Knezo CERN


Selected gridftp server

GridFTP

Control

Data

2811

GridFTP

process

GridFTP server

RFIO

CASTOR stager

Tapes

Selected GridFTP server

Globus Toolkit GridFTP-1.5 server

  • Based on wu-ftp 2.6.2

  • Widely used

    • expected good support

  • Supported GridFTP extensions:

    • EBLOCK mode

    • PARALLEL transfer

    • REST STREAM

    • DCAU

    • ERET, ESTO

  • Also supported:

    • Third-party transfer

    • PBSZ, PROT

    • MDTM

  • Not supported GridFTP extensions:

    • STRIPING, SPAS, SPOR

    • ABUF, SBUF

CASTOR & GridFTP / Emil Knezo CERN


Gridftp problems

GridFTP problems

  • Firewalls

    • Bi-directional data transfer in EBLOCK mode

      • Cannot open data-connection – blocked by firewall

    • Firewalls with NAT

      • GSI mutual authentication errors

  • HSM

    • Data existing in HSM name space are not always readily accessible:

      • Possible disconnection of idle control channel socket by some firewalls

      • Third-party transfer from HSM suffers from data-connection accept timeout at the data-receiving end.

  • Solution

    • Firewall:

      • Do not use firewalls with NAT

      • Do not block data-connections in firewall

    • HSM:

      • Always pre-stage your data in HSM before transfer

        • Currently with CASTOR “stagein” command; later when available with SRM interface.

CASTOR & GridFTP / Emil Knezo CERN


External network connection

GEANT

US-link

DataTAG

1Gb/s

622Mb/s

2.5Gb/s

router

350Mb/s half-duplex

PIX

HTAR

1Gb/s

350Mb/s half-duplex

router

1Gb/s

1Gb/s

GridFTP

server

External network connection

  • GridFTP data-connections to/from CASTOR GridFTP server are routed via 1Gb/s High Throughput Access Route (HTAR)

  • GridFTP control-connections are routed via PIX (TCP window size is fixed to 64kB if data-connection goes via PIX).

  • We share 1Gb/s link to GEANT,622 Mb/s connection to US institutes.

  • Only high # ports connections (data-connections) to/from CASTOR GridFTP server are routed via HTAR

    • Port #s interval currently applicable:<50k,51k>

    • Configuration issue

CERN

CASTOR & GridFTP / Emil Knezo CERN


Castor gridftp test service

CASTOR/GridFTP test-service

CERN

  • Test service in operation from mid-January 2003

  • Installation based on

    • EDG Globus, rel.24 (January – middle of June)

    • VDT 1.1.8 (since middle of June)

  • Supports

    • All EDG GridFTP clients, globus-url-copy

  • Still on server-code To-Do list

    • 64-bit file support (currently no files > 2GB)

    • CWD, CDUP fails on CASTOR name-space (“..” problem).In the meantime, full path is to be used by clients for CASTOR files

    • Internal “ls” to go fully rfio, at the moment CASTOR’s “nsls”client used

    • Test some GridFTP commands currently not used by supportedGridFTP clients (ESTO, ERET)

CASTOR

wacdr002d

GridFTP

1 Gbit/s GEANT link

1Gbit/s

(via HTAR since mid-May)

622 Mbit/s US link

rfio

GridFTP

CASTOR & GridFTP / Emil Knezo CERN


Evolution of castor gridftp service

CASTOR

stageatlas

Serv_1

griftpd

GridFTP via HTAR

cms001d

Serv_2

rfio

stagepublic

griftpd

DNSload-balancing

Serv_n

griftpd

UID – stager mapping

Evolution of CASTOR/GridFTP service

  • Set of configurations extended by UID--Stager mapping

  • DNS-load balancing (still to be verified)

  • Stager-response logging

  • Increased data-connection accept timeout (20 min)

CASTOR & GridFTP / Emil Knezo CERN


Performance and statistics

Performance and statistics

  • Performance

    • CERN internal transfer was: 5MB/s in/out; now: 7MB/s in/out

    • Transfer from NIKHEF was 3MB/s in/out; now: not available yet

      • Standard CERN TCP configuration (64kB TCP buffer size)

      • Not via HTAR

      • 10 parallel streams

  • Statistics

    • Not properly kept

      • Ftp-xferlog file – no file size for outbound traffic

      • GridFTP-xferlog – repeated file-record per every parallel stream of a transfer

    • Example: 2 weeks statistics May 26 – June 9:

      • Transferred 1480 files (1217 inbound, 263 outbound)

      • 627,425 GB stored to CASTOR via GridFTP wacdr002d service

      • Main user: ATLAS

        • gppui04.gridpp.rl.ac.uk, aftpexp.bnl.gov, lscf.nbi.dk

CASTOR & GridFTP / Emil Knezo CERN


Dn user mapping

DN -- User mapping

EDG-mechanisms used

  • grid-mapfile with mapping granularity on VO-level

    • Currently un-maintainable to have user-level mapping granularity

    • No dynamic pool accounts; edg-gridmap.conf:

      group ldap://grid-vo.nikhef.nl/ou=testbed1,o=alice,dc=eu-datagrid,dc=orgalice001 group ldap://grid-vo.nikhef.nl/ou=testbed1,o=atlas,dc=eu-datagrid,dc=orgatlas001 group ldap://grid-vo.nikhef.nl/ou=tb1users,o=cms,dc=eu-datagrid,dc=orgcms001 group ldap://grid-vo.nikhef.nl/ou=tb1users,o=lhcb,dc=eu-datagrid,dc=orglhcb001 group ldap://grid-vo.nikhef.nl/ou=tb1users,o=biomedical,dc=eu-datagrid,dc=orgbiome001 group ldap://grid-vo.nikhef.nl/ou=tb1users,o=earthob,dc=eu-datagrid,dc=orgob001 group ldap://marianne.in2p3.fr/ou=ITeam,o=testbed,dc=eu-datagrid,dc=orgiteam001 group ldap://marianne.in2p3.fr/ou=wp6,o=testbed,dc=eu-datagrid,dc=orgwpsix001

    • Up to VO Admin to create subsets of users (new LDAP URLs) for other UIDs

  • One DN – One User restriction

    • Hard to sell to experiments

    • VOMS should solve the problem

      • VOMS provide <DN + role> based UID mapping

      • VOMS to be tested with CASTOR GridFTP server (configuration issue)

CASTOR & GridFTP / Emil Knezo CERN


Umask and usage examples

Umask and usage examples

  • Umask 002 => “rw-rw-r—” permissions on CASTOR

    • Per server umask configuration

    • CASTOR at the moment still requires world-readable files

  • Usage examples

    • Prestage file

      stagein [-h wacdr002d] -M /castor/cern.ch/atlas/subdirectory/file.name

      (will be replaced by SRM call)

    • Retrieve file from CASTOR

      globus-url-copy [-p 10]gsiftp://wacdr002d.cern.ch/castor/cern.ch/atlas/subdirectory/file.namefile:///home/knezo/file.name

    • Third party transfer from CASTOR

      globus-url-copy [-p 10]gsiftp://wacdr002d.cern.ch/castor/cern.ch/atlas/subdirectory/file.namegsiftp://spider.usatlas.bnl.gov/usatlas/workarea/knezo/file.name

    • Directory listing

      edg-gridftp-ls –verbosegsiftp://wacdr002d.cern.ch/castor/cern.ch/atlas/

CASTOR & GridFTP / Emil Knezo CERN


Plan for castor gridftp service

Plan for CASTOR/GridFTP service

  • One year horizon

    • Support for CMS world-wide production

      • This is now High Priority Task

      • Performance challenge for server

        • Requires TCP-tuning, likely dedicated stager, maybe NAPI

    • DNS load-balanced cluster of GridFTP servers

      • Sufficient for users with no strict throughput requirements for the coming year (ATLAS, LHCB, EDG)

    • Service To-Do list

      • Performance tuning

      • DNS-load balancing configuration tests

      • Prepare user & admin documentation, plus rpms

        • Shown interest from external institutes: INFN, IFAE, IFIC

      • Integrate with CERN monitoring, plus scripts to create server usage statistics

      • Still to improve logging

      • Synchronisation on package upgrades with EDG

      • VOMS to improve DN–User mapping

  • Beyond one year

    • Need to understand what the Globus GridFTP server evolution will be.

CASTOR & GridFTP / Emil Knezo CERN


Conclusions

Conclusions

  • GridFTP interface to CASTOR already exists

  • Ready to use service requires to solve:

    • Configuration issues

    • Performance issues

    • Admin issues

  • CASTOR/GridFTP service has potential to satisfy CASTOR users for a year

CASTOR & GridFTP / Emil Knezo CERN


  • Login