1 / 19

Scalla/xrootd Introduction

Scalla/xrootd Introduction. Andrew Hanushevsky, SLAC SLAC National Accelerator Laboratory Stanford University 6-April-09 ATLAS Western Tier 2 User’s Forum. Outline. File servers NFS & xrootd How xrootd manages file data Multiple file servers (i.e., clustering)

nathan
Download Presentation

Scalla/xrootd Introduction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Scalla/xrootd Introduction Andrew Hanushevsky, SLAC SLAC National Accelerator Laboratory Stanford University 6-April-09 ATLAS Western Tier 2 User’s Forum

  2. Outline • File servers • NFS & xrootd • How xrootd manages file data • Multiple file servers (i.e., clustering) • Considerations and pitfalls • Getting to xrootd hosted file data • Available programs and interfaces 2

  3. NFS Server NFS Client Data Files Application xroot Server xroot Client Linux Linux Client Machine Server Machine File Server Types Application Alternatively Data Files Linux Linux Client Machine Server Machine xrootd is nothing more than an application level file server & client using another protocol 3

  4. Why Not Just Use NFS? • NFS V2 & V3 inadequate • Scaling problems with large batch farms • Unwieldy when more than one server needed • NFS V4? • Relatively new • Multiple server support still being vetted • Still has a single point of failure problems 4

  5. Application NFS Server NFS Server Data Files Data Files NFS Client Linux Linux Linux Client Machine Server Machine A Server Machine B NFS & Multiple File Servers cp /foo /tmp open(“/foo”); Which Server? NFS can’t naturally deal with this problem. Typical ad hoc solutions are cumbersome, restrictive and error prone! 5

  6. xroot Server xroot Server Linux Linux Server Machine A Server Machine R 1 Who has /foo? Try B 3 4 open(“/foo”); 2 I do! xrootd & Multiple File Servers I Data Files xrdcp root://R//foo /tmp Application xroot Client Redirector open(“/foo”); Linux Client Machine /foo Data Files xroot Server The xroot client does all of these steps automatically without application (user) intervention! Linux Server Machine B 6

  7. File Discovery Considerations I • The redirector does not have a catalog of files • It always asks each server, and • Caches the answers in memory for a “while” • So, it won’t ask again when asked about a past lookup • Allows real-time configuration changes • Clients never see the disruption • Does have some side-effects • The lookup takes less than a microsecond when files exist • Much longer when a requested file does not exist! 7

  8. xroot Server xroot Server Linux Linux Server Machine R Server Machine A 1 Who has /foo? Nope! 2 xrootd & Multiple File Servers II Data Files xrdcp root://R//foo /tmp Application xroot Client Redirector open(“/foo”); Linux Client Machine 5 /foo Data Files xroot Server File deemed not to exist if there is no response after 5 seconds! Linux Server Machine B 8

  9. File Discovery Considerations II • System optimized for “file exists” case! • Penalty for going after missing files • Aren’t new files, by definition, missing? • Yes, but that involves writing data! • The system is optimized for reading data • So, creating a new file will suffer a 5 second delay • Can minimize the delay by using the xprep command • Primes the redirector’s file memory cache ahead of time • Can files appear to be missing any other way? 9

  10. Missing File vs. Missing Server • In xroot files exist to the extent servers exist • The redirector cushions this effect for 10 minutes • Afterwards, the redirector cannot tell the difference • This allows partially dead server clusters to continue • Jobs hunting for “missing” files will eventually die • But jobs cannot rely on files actually being missing • xroot cannot provide a definitive answer to “" s: Ø$x” • This requires manual safety for file creation 10

  11. Safe File Creation • Avoiding the basic problem.... • Today’s new file may be on yesterday’s dead server • Generally, do not re-use output file names • Otherwise, serialize file creation • Use temporary file names when creating new files • E.g., path/....root.temp • Remove temporary to clean-up any previous failures • E.g., -f xrdcp option or truncate option on open • Upon success, rename the temporary to its permanent name 11

  12. Getting to xrootd hosted data • Use the root framework • Automatically, when files named root://.... • Manually, use TXNetFile() object • Note: identical TFile() object will not work with xrootd! • xrdcp • The copy command • xprep • The redirector seeder command • Via fuse on atlint01.slac.stanford.edu • POSIX preload library 12

  13. Copying xrootd hosted data • xrdcp [options] sourcedest • Copies data to/from xrootd servers • Some handy options: • -f erase dest before copying source • -s stealth mode (i.e., produce no status messages) • -Sn use n parallel streams (use only across WAN) 13

  14. Preparing xrootd hosted data • xprep [options] host[:port] [path [...]] • Prepares xrootd access via redirector host:port • Minimizes wait time if you are creating many files • Some handy options: • -w file will be created or written • -ffn file fn holds a list of paths, one per line 14

  15. Interactive xrootd hosted data • Atlas xroot redirector mounted as a file system • “/xrootd” on atlint01.slac.stanford.edu • Use this for typical operations • dq2-get • dq2-put • dq2-ls • rm 15

  16. For Everything Else • POSIX preload library (libXrdPosixPreload.so) • Works with any POSIX I/O compliant program • Provides direct access to xrootd hosted data • Does not need any changes to the application • Just run the binary as is • Talk to Wei or Andy if you want to use it 16

  17. Conclusion • We hope that this is an effective environment • Production • Analysis • But, we need your feedback • What is unclear • What is missing • What is not working • What can work even better 17

  18. Future Directions • More simplicity! • Integrating thecnsdintocmsd • Reduces configuration issues • Pre-linking the extended open file system (ofs) • Less configuration options • Tutorial-like guides! • Apparent need as we deploy at smaller sites 18

  19. Acknowledgements • Software Contributors • Alice: Derek Feichtinger • CERN: Fabrizio Furano , Andreas Peters • Fermi: Tony Johnson (Java) • Root: Gerri Ganis, Beterand Bellenet, Fons Rademakers • STAR/BNL: Pavel Jackl • SLAC: Jacek Becla, Tofigh Azemoon, Wilko Kroeger • LBNL: Alex Sim, Junmin Gu, Vijaya Natarajan (BeStMan team) • Operational Collaborators • BNL, FZK, IN2P3, RAL, UVIC, UTA • Partial Funding • US Department of Energy • Contract DE-AC02-76SF00515 with Stanford University 19

More Related