New kanga export scheme
This presentation is the property of its rightful owner.
Sponsored Links
1 / 11

New KanGA Export Scheme PowerPoint PPT Presentation


  • 87 Views
  • Uploaded on
  • Presentation posted in: General

New KanGA Export Scheme. Tim Adye Rutherford Appleton Laboratory BaBar Collaboration Meeting Data Distribution Session 13 th December 2000. ftp TNG Selecting events for export Running the export procedure. Optimising ftp throughput.

Download Presentation

New KanGA Export Scheme

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


New kanga export scheme

New KanGA Export Scheme

Tim Adye

Rutherford Appleton Laboratory

BaBar Collaboration Meeting

Data Distribution Session

13th December 2000

Tim Adye


New kanga export scheme

  • ftp TNG

  • Selecting events for export

  • Running the export procedure

Tim Adye


Optimising ftp throughput

Optimising ftp throughput

  • ftp / scp / rsync(syncslac) are not optimised for bulk transfers over the WAN

  • Optimisations possible

    • Use multiple-streams

      • Separate processes/threads copy different parts of the data

    • Adjust TCP/IP parameters

      • Large improvements from increasing TCP window size

  • Tools: bbftp, sfcp, grid-ftp (new version of gsi-ftp)

Tim Adye


New ftp tools

New “ftp” tools

  • bbftp[Gilles Farrache, IN2P3]

    • Retry mechanism

    • On-the-fly compression

    • Now with ssh authentication option [TJA]

      • Greatly simplifies automatic operation

      • Allows access to [email protected]

  • sfcp[Artem Trunov and Andy Hanushevsky, SLAC]

    • Better control of transfer parameters

    • Simple setup, but Solaris only

    • No compression, retry, etc.

Tim Adye


New ftp tools1

New “ftp” tools

  • grid-ftp (gsi-ftp) [GLOBUS]

    • GLOBUS authentication

    • Not yet publicly available

Tim Adye


Transfer performance

Transfer performance

  • SLAC->RAL Kanga exports

    • syncslac / rsync gave ~300 kbits/s per job

      • Scales with number of parallel copies

    • bbftp with 2 streams gives 3-6 Mbits/s

      • Nearing 10 Mbits/s Janet-ESnet peering

    • sfcp with 2 streams gives 600 kbits/s

      • Scales with number of streams

      • Need to understand sfcp/bbftp differences

  • RAL->SLAC SP imports

    • bbftp gives 10-20 Mbits/s

      • Benefits from x3 compression of Objy files

Tim Adye


Caveat

Caveat

  • The optimal bbftp/sfcp choice/settings will vary from site to site.

    • Rome->RAL: bbftp and sfcp are similar

    • Eg. SLAC<->INFN gain with larger number of streams

  • Need systematic tests to give better advice

Tim Adye


Kanga export procedures

Kanga export procedures

  • Kanga export controlled by skimData database

    • David Kirkby, Alessandra Forti, Alvise Dorego,TJA

  • skimSqlMirror makes local copy of skimData db

  • skimSqlSelectselects data for import

  • skimImportcopies the selected files

  • skimBackupbackup/archive to tape

  • skimDeleteremove files from local disk

  • Documentation under construction at

    • http://www.pd.infn.it/~dorigoa/BFROOT/KDDIBT.html

    • BaBar -> Computing -> Data Dist -> skimData remote

Tim Adye


Skimsqlselect

skimSqlSelect

  • Selects data for import by setting a flag in skimData database

  • Run as part of skimData mirroring to automatically select new data

  • Use skimData options to select data, eg.

    • skimSqlSelect

      -–skimData ‘-sAllEventsKanga:-S>=Skim880f’

Tim Adye


Skimimport

skimImport

  • Copies files that are marked for import in the skimData database

  • Can use different ftp programs

    • Currently implements bbftp and sfcp

    • Plan to add rsync which can check existing files

    • Perl OO interface can be used by other scripts

      • Eg. Automatic SP export procedure

  • Can distribute files across different filesystems.

  • Eg.

    • skimImport --ftp-type=bbftp –config=ral.cfg

Tim Adye


Deployment

Deployment

Need guinea pigs to try this all out

Tim Adye


  • Login