1 / 32

WebFTS File Transfer Web Interface for FTS3

WebFTS File Transfer Web Interface for FTS3. Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing. Overview. The FTS3 service WebFTS features WebFTS cloud integrations Dropbox CERNBox Ongoing development Dropping X509

pomona
Download Presentation

WebFTS File Transfer Web Interface for FTS3

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. WebFTSFile Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing

  2. Overview • The FTS3 service • WebFTS features • WebFTS cloud integrations • Dropbox • CERNBox • Ongoing development • Dropping X509 • Data management operations WebFTS: File Transfer Web Interface for FTS3

  3. What is WebFTS? • Web based tool to transfer files between grid/cloud storages • Modular protocol support • gsiftp, http(s), xrootd and srm • Cloud extensions: dropbox, CERNBox • Development funded by WebFTS: File Transfer Web Interface for FTS3

  4. Provide access to leading technology Based on FTS3 • FTS3 is the service responsible for distributing the majority of LHC data across the WLCG infrastructure • Low level data movement service, responsible for moving sets of files from one site to another while allowing participating sites to control the network resource usage • Used by LHC VOs + many others VOs part of EGI • ~20PB monthly transfer volume / ~2.2M files per day (WLCG) http://dashb-fts-transfers.cern.ch/ui/ WebFTS: File Transfer Web Interface for FTS3

  5. WebFTS architecture BROWSER WEBFTS.js REST API REST API FTS3 GFAL2 DAVIX GSIFTP DROPBOX … WebFTS: File Transfer Web Interface for FTS3

  6. Security • Simpler access while keeping the same level of security • X509 + Oauth for Dropbox • Transparent delegation of credentials • Avoid storing or transferring any sensitive data • Open access to all source code • All sensitive information is used within the browser and forgotten WebFTS: File Transfer Web Interface for FTS3

  7. Delegation • Delegation is needed to let WebFTS access the grid on users behalf • Users make private key available to browser • Not available via browser API • VOMS extensions acquired by the service on users behalf • Why it’s important • Gives the users a service which can access the grid for them, from a browser, with full VOMS credentials WebFTS: File Transfer Web Interface for FTS3

  8. Additional Features • Check-summing and file overwriting • Possibility to resubmit transfer jobs or only-failed files transfers. • Storage Endpoints Auto-completion • For endpoints published on the BDII (EGI and WLCG Information System) • Support for LFC Registration • File catalog developed at CERN and used by EGI and WLCG WebFTS: File Transfer Web Interface for FTS3

  9. Success Stories • WebFTS has been successfully tested to transfer from/to: • EUDAT B2Stage ( iRODS DSI) • Any gsiftp/webdav/xrootd aware grid storage ( DPM, dCache, Castor, EOS, Storm) • HPC Titan @ Oak Ridge National Lab (ongoing) • https://www.olcf.ornl.gov/titan/ • Under evaluation by LHCb WebFTS: File Transfer Web Interface for FTS3

  10. Landing page and Guided-tour WebFTS: File Transfer Web Interface for FTS3

  11. Credential delegation ZERO SENSITIVE INFORMATION IS TRANSMITTED WebFTS: File Transfer Web Interface for FTS3

  12. Transfer interface WebFTS: File Transfer Web Interface for FTS3

  13. Job status interface WebFTS: File Transfer Web Interface for FTS3

  14. Extension for Dropbox • Nice way import/export data from the grid world • Avoid the installation of new software and uses what the user has already installed • Zero development of clients • Multiplatform is given for free • Integration with Oauth • By delegating to FTS the right to interact with dropbox on users behalf • Achieved using web tech • Which requires the interactivity of a browser WebFTS: File Transfer Web Interface for FTS3

  15. Extension for Dropbox WebFTS: File Transfer Web Interface for FTS3

  16. Dropbox plugin • Server side the development of a plugin for the metadata management and I/O operations was needed: • FTS REST integrates the plugin to perform metadata management operations • FTS3 server uses the plugin to perform the transfers: • GridFTP <-> dropbox • Http(s) <-> dropbox WebFTS: File Transfer Web Interface for FTS3

  17. CERNBox integration • While Dropbox has been integrated via the implementation of a plugin for CERNBox we waited for the new version with EOS as backend ( CERNBox 2.0) • We use EOS access via standard grid protocols ( e.g. xrootd) • We map user credentials to correct EOS namespace • The rest comes for free WebFTS: File Transfer Web Interface for FTS3

  18. WebFTS With CERNBox WebFTS: File Transfer Web Interface for FTS3

  19. WebFTS With CERNBox WebFTS: File Transfer Web Interface for FTS3

  20. WebFTS With CERNBox WebFTS: File Transfer Web Interface for FTS3

  21. WebFTS With CERNBox WebFTS: File Transfer Web Interface for FTS3

  22. WebFTS With CERNBox WebFTS: File Transfer Web Interface for FTS3

  23. Ongoing developments:Access without X509 How can we get rid of the delegation step? • An Identity Federation: eduGAIN • To allow identity providers to authenticate users at their own institute (SSO) • A token translation service : STS • To ask the CA for a certificate for the users • An “IOTA” Certification Authority • To grant the short lived certificate • VOMS • To accept the new cert as a VO member WebFTS: File Transfer Web Interface for FTS3

  24. EDUGAIN • Built on existing federations and infrastructures • CERN participates in eduGAIN via SWITCHaai • Many NRENs participate in eduGAIN too WebFTS: File Transfer Web Interface for FTS3 24

  25. Security Token Service (STS) An EMI service SAML in, X509/VOMS out WebFTS: File Transfer Web Interface for FTS3

  26. “IOTA” CA WebFTS: File Transfer Web Interface for FTS3

  27. VOMS admin WebFTS: File Transfer Web Interface for FTS3

  28. Architecture IOTA CA STS IdP IdP IdP IdP CERN SSO VOMS X.509 VOMS SAML SAML Redirect WAYF Credentials Attributes Grid Storage Element WebFTS X.509 VOMS Web Slide adapted from Romain Wartel, GDB Sept 2014

  29. Pros/Cons • X509-free access to the grid infrastructure • With VOMS support • Without modifying all the services • Federated single sign on • One password to remember • Numerous services potentially accessible • But we need Site acceptance WebFTS: File Transfer Web Interface for FTS3

  30. Ongoing developments:Data Management • Not only Transfers.. • FTS REST API have been extended to support data management operations • Delete • Create/Remove folders • Rename • Under integration in WebFTS WebFTS: File Transfer Web Interface for FTS3

  31. Links • Online service accessible: • https://webfts.cern.ch try now! • User certificate in your browser • User guide, F.A.Q: • Online guided-tour • http://fts3-service.web.cern.ch/documentation/webfts • Official support & code • fts-support@cern.ch • https://github.com/cern-it-sdc-id/webfts WebFTS: File Transfer Web Interface for FTS3

  32. Questions? ideas use cases feedback fts-support@cern.ch WebFTS: File Transfer Web Interface for FTS3

More Related