data transfer efficiency leave no byte unchurned n.
Skip this Video
Loading SlideShow in 5 Seconds..
Data Transfer Efficiency - leave no byte unchurned PowerPoint Presentation
Download Presentation
Data Transfer Efficiency - leave no byte unchurned

Loading in 2 Seconds...

  share
play fullscreen
1 / 24
Download Presentation

Data Transfer Efficiency - leave no byte unchurned - PowerPoint PPT Presentation

aminia
73 Views
Download Presentation

Data Transfer Efficiency - leave no byte unchurned

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. Data Transfer Efficiency- leave no byte unchurned Jens Jensen Rutherford Appleton Laboratory GridPP26, U Sussex, March 2011

  2. Background • GridPP’sdata grid • Distributed Storage Elements • Data movers (FTS, PhEDExet al) • Catalogues (usu. replica) • e-Infrastructure (aka cyberinfrastructure) • (Presentation at ISGC)

  3. The Data Grid • WLCG is primarily a data grid • Computation can (in principle) be redone • Jobs go to where data is • Moving a job is quicker than moving data

  4. Premature Optimisation is the Root of All Evil

  5. Postmature non-optimisation is the root of some evil • The role of infrastructure code • Scientist as a programmer • “Bad” code moves up the stack? • “Bad” code improves over time? • Doofers stay in prod’n

  6. Efficiencaciousness Goals Service • Availability • Performance • Grows as needed • Robust (no SPoF?) People • (Effective) support • Training • Expertise • Availability of…

  7. Approaches • Philosophy • Get it done – WLCG • Get it done right – EGI? • Do It Perfectly The First Time… • Evolutionary (control system) vs revolutionary • Proactive vs reactive

  8. Efficiencaciousness Issues • Failures • Sites – BDII, network • Elements – storage • Components – disk servers • Timeouts • DDoS

  9. Efficiencaciousness Issues • Overall effort • Funded, contributed, external • Availability of expertise • Single Point of Knowledge • Decoherence • 2nd Law of Thermodynamics • Learning from incidents

  10. Efficiencaciousness Issues • Primary communication • Sites • Users: large VOs, small VOs, single users • PMB • Secondary • WLCG • NGS

  11. Efficiencaciousness Issues • Sites • There Is Always A Bottleneck Somewhere • Site dependent • Usage dependent • Information • Freshness • Accuracy (“spped is substutefoaccurcy”)

  12. Efficiencaciousness Issues • Usage patterns • C.f. Wahid’s talk yesterday • WAN vs LAN (WN) traffic • Technology • In the narrow sense (drives, controllers) • And the wider sense: dist’dfilesystems • Support: Upstream (EGI), Fabric

  13. Efficiencaciousness Issues • Overheads • Complexity of use of stack (see next) • Infrastructure is complex • But Complexity Has To Go Somewhere • Time-to-production • Testing, troubleshooting, monitoring, tweaking, tuning

  14. With apologies to the OSI stack

  15. Particular Pain Point Principle Progress

  16. Progressing Forward • What is progress • How to measure progress

  17. The Good News • We’ve come a long way • Don’t think there is a skills gap • But some SPoKs

  18. Graeme’s talk • “Get the best out of what we can afford to buy” • Proactive sites better • Standards are good

  19. E[GM]I involvement • EMI data roadmap • Support for dCache, DPM, StoRM • Support for standards (NFS4, CDMI) • But then • StoRM=INFN, dCache=DESY, DPM=CERN

  20. The Cloud View • Supplement resources with on-demand • Agile • CDMI is superset of SRM • But using ReST+JSON, not SOAP

  21. (Open) Standards • Standards promote interoperation and stability • Interoperation • Multiple (independent) implementations • Both Java and (C or C++)

  22. The Case for Non-HEP Data • Benefit from non-HEP data • Outreachy stuff • Benefit to society (eg saving lives) • NGI interop (at compute) • Others…

  23. Summary

  24. Efficiencaciousness Goals Service • Availability • Performance • Grows as needed • Robust (no SPoF?) People • (Effective) support • Training • Expertise • Availability of…