1 / 29

xrootd

xrootd. Andrew Hanushevsky Stanford Linear Accelerator Center 30-May-03. Goals. High Performance File-Based Access Scalable, extensible, usable Fault tolerance Server failures handled in a natural way Servers may be dynamically added and removed Flexible Security

kyria
Download Presentation

xrootd

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. xrootd Andrew Hanushevsky Stanford Linear Accelerator Center 30-May-03

  2. Goals • High Performance File-Based Access • Scalable, extensible, usable • Fault tolerance • Server failures handled in a natural way • Servers may be dynamically added and removed • Flexible Security • Allowing use of almost any protocol • Rootd Compatibility 2: xrootd

  3. Achieving High Performance • Scalable request/response protocol • Multi-threaded multi-process architecture • Architecture sensitive polling • MRU scheduling • Sticky sockets • Adaptive reconfiguration • Versatile sfs layer (based on proven oofs) 3: xrootd

  4. Scalable Protocol I • Connection multiplexing • One connection per client/host • Multiple logically independent streams • Request redirection supported • Similar to http redirection • Supports dynamic load balancing and fail-over • Uses an intentional request header • Can better optimize request processing 4: xrootd

  5. Scalable Protocol II • Asynchronous mode allowed • Multiple processing-order-independent requests • Optional application-directed pre-read • I/O segmenting • Able to naturally deal with very large transfers • Better use of server resources • Request deferral • Client waits for resources without using server resources 5: xrootd

  6. Scalable Protocol III • Unsolicited Reverse Request Mode • Allows server to manage client for recovery • Asynchronous redirect, deferral, and messages • Protocol may be compatibly extended • Mechanism to send opaque information • Accommodate things that were “forgotten” • Messaging interface • Cache group • Request priority • And so on…. 6: xrootd

  7. MT/MP Architecture • Normally one multi-threaded server per host • Should be able to utilize available resources • Easy to administer • Optionally, multiple servers per host • Fully utilize large machines 7: xrootd

  8. Architecture Sensitive Polling • All POSIX systems support poll() • Used by default • Not always an efficient I/O “interrupt” mechanism • Alternate polling mechanisms allowed • /dev/poll • Available on Solaris and patched RH Linux • Up to an order of magnitude reduction in CPU • Essential to reduce latency 8: xrootd

  9. MRU Scheduling • Connections processed in most recently used order • Gives priority to active connections • Reduces polling overhead • Essentially a fair scheduling algorithm • Starvation cannot occur • Longer running tasks tend to get started first • Assuming all other things being equal 9: xrootd

  10. Sticky Sockets • Connection temporarily binds to a thread • Avoids polling and scheduling overhead • Significantly reduces latency • Connection automatically unbinds • Client is not sufficiently active • Number of other requests approaches available threads 10: xrootd

  11. Adaptive Reconfiguration • Server dynamically adjusts configuration • Number of threads • Kept proportionate to number of active requests • Pre-allocated buffers • Sizes track actual usage profile • Recomputed periodically • Pre-allocated objects • Number tracks recent needs • High latency connections rescheduled 11: xrootd

  12. Versatile sfs Layer I • Integrates multiple performance features • Dynamic load balancing • Client redirected to “best” server of the moment • File descriptor partitioning • Reduces socket polling overhead • File system interface reuse • Prevents open file proliferation and attendant overhead • Same file opened in same mode shared by multiple clients • File system interface timeout • Reduces overhead caused by idle opened files 12: xrootd

  13. ufs ufs ufs Dynamic Load Balancing xrootd Dynamic Selection xrootd mss client xrootd 13: xrootd

  14. xrootd xrootd xrootd (any number) dlbd dlbd dlbd dlbd (any number) xrootd DLB Implementation I do subscribe who has the file? open again open Client wait try host:port 14: xrootd

  15. Versatile sfs Layer II • Dynamic disk cache integration • Allows unlimited file system size • Provides superior internal load balancing • Mass Storage Integration • HPSS, Castor, Enstore, etc • RFIO Integration • Scalable authorization • From file sub-trees to single files 15: xrootd

  16. Cache File System Index Area Optional data cache Default data area /databases/mydbfile symlink Naming convention allows for audit and index recovery Multiple Independent Filesystems /cache1/databases:mydbfile Data Area Any number Any Size Chosen based on free space in LRU order /cache2 /cache3 16: xrootd

  17. Fault Tolerance I • Servers may come and go • Uses load balancing to effect recovery • New servers can be added at any time • Servers may be brought down for maintenance • Files can be moved around in real-time • Client simply adjust to the new configuration • XTNetFile object handles recovery protocol 17: xrootd

  18. Fault Tolerance II • Whenever client looses r/o connection • Back to distinguished xrootd(s) for reselection • Whenever client looses r/w connection • Limited wait/retry loop on the same server • We will be working to improve this next year! • All handled in the XTNetFile class • Disruptions merely delay the client 18: xrootd

  19. Flexible Security • Negotiated Security Protocol • Allows client/server to agree on protocol • E.g., Kerberos, GSI, AFS Kerberos, etc. • Can be easily extended • Multi-protocol authentication support 19: xrootd

  20. Security Token Security Architecture client xrootd login Client-Specific Security Configuration authenticate Multiple handshakes allowed during authentication phase (required by some PKI protocols) Protocol Selection Self Configuration Security Shared Library Security Shared Library libooseccl.so libooseccl.so 20: xrootd

  21. Heterogeneous Security Support xrootd 1 Security Contexts •Servers have one or more protocol objects •Server protocol objects created at server initialization time •Client selects which protocol to use when security context created •Protocol object created based on configuration returned by xrootd •One security context object per physical xrootd connection •Protocol objects may be shared by one or more contexts •Each “pass” through a security context object may generate credentials to be passed to xrootd xrootd 2 client xrootd 3 protocols Protocol Objects 21: xrootd

  22. Simple & Effective Interface • For each login that requires authentication • XrdSecCreateSecurityContext(ipaddr, config) • Returns security protocol object • XrdSecClientSecurity • Based on server ipaddr and server-supplied config • XrdSecClientSecurity::getCredentials() • Returns credentials to be sent to the server • Done via authenticate request and possible authmore response • Based on well tested and documented oofs security 22: xrootd

  23. libooseccl.so Security Shared Library Optional Scalable Authorization libooacc.so xrootd Authentication Authorization Shared Library Authorization XrdSfs Replaceable Database Interface database u abh rw /slac/rootfiles/usr/abh r /cern/rootfiles 23: xrootd

  24. Security Summary • Multi-protocol Authentication • Supports distributed heterogeneous environments • Scalable Authorization • Open-ended capability based model • Integrated Auditing • To keep the security hard hats happy • Well defined, proven interfaces • Trivially replaceable for a plug & play architecture 24: xrootd

  25. rootd Compatibility • Bilateral compatibility • XTNetfile reverts to TNetFile for rootd servers • XRootd reverts to rootd protocol for TNetFile • Allows for transparent introduction • Can run mixed mode • Binary is multi-environment compatible 25: xrootd

  26. Compatibility Modes Client-Side Compatibility Application XTNetFile xrootd rootd TNetFile Server-Side Compatibility Application rootd xrootd rootd compability TNetFile 26: xrootd

  27. xrootd Architecture application Protocol Manager xrd Protocol Layer xroot Authentication xrootd Filesystem Logical Layer sfs Authorization Filesystem Physical Layer oss Filesystem Implementation _fs mss 27: xrootd

  28. xrootd Internals network manager config manager socket manager thread manager buffer manager protocol manager xroot rootd security Dynamically loaded (can also be static) oofs logging 28: xrootd

  29. Conclusion • xrootd provides high performance file access • Improves over afs, ams, nfs, etc. • Unique performance, usability, scalability, security, compatibility, and recoverability characteristics • xrootd can provide a firm server foundation for native file system implementations • E.g. alienfs, gridfs, slashgrid, etc • For now, aim is to support BaBar 29: xrootd

More Related