The next generation root file server
Download
1 / 24

The Next Generation Root File Server - PowerPoint PPT Presentation


  • 110 Views
  • Uploaded on

The Next Generation Root File Server. Andrew Hanushevsky Stanford Linear Accelerator Center 27-September-2004 http://xrootd.slac.stanford.edu. What is xrootd?. File Server Provides high performance file-based access Scalable, extensible, naively usable Fault tolerant

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' The Next Generation Root File Server' - radley


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
The next generation root file server

The Next Generation Root File Server

Andrew Hanushevsky

Stanford Linear Accelerator Center

27-September-2004

http://xrootd.slac.stanford.edu


What is xrootd
What is xrootd?

  • File Server

    • Provides high performance file-based access

    • Scalable, extensible, naively usable

    • Fault tolerant

      • Server failures handled in a natural way

      • Servers may be dynamically added and removed

    • Secure

      • Framework allows use of almost any protocol

    • Rootd Compatible

2: xrootd


Goals ii
Goals II

  • Simplicity

    • Can run xrootd out of the box

      • No config file needed for non-complicated/small installations

  • Generality

    • Can configure xrootd for ultimate performance

      • Meant for intermediate to large-scale sites

3: xrootd


How is high performance achieved
How Is high performance achieved?

  • Rich but efficient server protocol

    • Combines file serving with P2P elements

    • Allows client hints for improved performance

      • Pre-read, prepare, client access & processing hints,

    • Multiplexed request stream

      • Multiple parallel requests allowed per client

  • An extensible base architecture

    • Heavily multi-threaded

      • Clients are dedicated threads whenever possible

    • Extensive use of OS I/O features

      • Async I/O, device polling, etc.

    • Load adaptive reconfiguration.

4: xrootd


Xrootd server architecture
xrootd Server Architecture

p2p heart

application

Protocol & Thread Manager

xrd

xrootd

Protocol Layer

xroot

Authentication

Filesystem Logical Layer

ofs

odc

Authorization

optional

Filesystem Physical Layer

oss

(included in

distribution)

Filesystem Implementation

_fs

mss

5: xrootd


Rootd bilateral compatibility
Rootd Bilateral Compatibility

Client-Side Compatibility

Application

TXNetFile

xrootd

rootd

TNetFile

Server-Side Compatibility

Application

rootd

xrootd

rootd compability

TNetFile

client

6: xrootd


How performant is it
How performant is it?

  • Can deliver data at disk speeds (streaming mode)

    • Assuming good network & proper TCP buffer size

  • Low CPU overhead

    • 75% less CPU than NFS for same data load

      • It is memory hungry, however.

  • General requirements

    • Middling speed machine

      • The more CPU’s the better

    • 1-2GB of RAM

7: xrootd


How is scalability achieved
How is scalability achieved?

  • Protocol allows server scalability

    • Server directed I/O segmenting

    • Request deferral to pace client load

    • Unsolicited responses for ad hoc client steering

    • P2P elements for lashing servers together

      • Request redirection key element

      • Integrated with a P2P control network

        • olbd servers provide control information

8: xrootd


How does it scale
How does it scale?

  • xrootd scales in multiple dimensions

    • Can run multiple load balanced xrootd’s

      • Provides single uniform name and data space

      • Scales from 1 to over 32,000 cooperating data servers

      • Architected as self-configuring structured peer-to-peer (SP2) data servers

      • Servers can be added & removed at any time

    • Client (TXNetFile) understands SP2 configurations

      • xrootd informs client when running in this mode

      • Client has more recovery options in the event of failure

9: xrootd


Load balancing implementation
Load Balancing Implementation

  • Control Interface (olbd)

    • Load balancing meta operations

      • Find files, change status, forwarded requests

  • Data Interface (xrootd)

    • Data is provided to clients

    • Interfaces to olbd via the ofs layer

  • Separation is important

    • Allows use of any protocol

    • Client need not know the control protocol

10: xrootd


Entities relationships

data

Entities & Relationships

xrootd

Data Network

(redirectors steer clients to data

Data servers provide data)

olbd

Control Network

Managers & Servers

(resource info, file location)

Redirectors

olbd

M

ctl

xrootd

olbd

S

Data Clients

xrootd

Data Servers

11: xrootd


Typical sp 2 configuration

ufs

ufs

Typical SP2 Configuration

subscribe

olbd

xrootd

Dynamic

Selection

olbd

xrootd

mss

client

redirector

xrootd

subscribe

olbd

12: xrootd


Example slac configuration
Example: SLAC Configuration

kan01

kan02

kan03

kan04

kanxx

kanolb-a

bbr-olb03

bbr-olb04

client machines

13: xrootd


Why do this
Why do this?

  • Can transparently & incrementally scale

  • Servers can come and go

    • Load balancing effects recovery

      • New servers can be added at any time

      • Servers may be brought down for maintenance

      • Files can be moved around in real-time

  • Client simply adjust to the new configuration

    • TXNetFile object handles recovery protocol

14: xrootd


What we have seen
What we have seen

  • For a single server:

    • 1,000 simultaneous clients

    • 2,200 simultaneous open files

  • Bottlenecks

    • Disk I/O (memory next behind)

15: xrootd


What have we heard
What Have We Heard

  • The system is too stable

    • Users run extra-long jobs (1-2 weeks) now

      • Error not discovered until weeks later

  • The system is too aggressive

    • New servers are immediately taken over

      • Easy configuration but startling for administrators

16: xrootd


Next getting remote data
Next: Getting remote data

SLAC

Firewall

Firewall

Firewall

IN2P3

RAL

IN2P3proxy

RALproxy

xrootd’s

Firewalls require

Proxy servers

17: xrootd


Proxy service
Proxy Service

  • Attempts to address competing goals

    • Security

      • Deal with firewalls

    • Scalability

      • Administrative

      • Configuration

    • Performance

      • Ad hoc forwarding for near-zero wait time

      • Intelligent caching in local domain

18: xrootd


Proxy implementation
Proxy Implementation

  • Uses capabilities of olbd and xrootd

    • Simply an extension of local load balancing

    • Implemented as a special file system type

      • Interfaces in the ofs layer

      • Functions in the oss layer

  • Primary developer is Heinz Stockinger

19: xrootd


Proxy interactions
Proxy Interactions

data01

data02

data03

data04

RAL

local olb

proxy olb

4

5

local olb

proxy olb

SLAC

3

1

red01

data02

data03

proxy01

2

client machines

20: xrootd


Why this arrangement
Why This Arrangement?

  • Minimizes cross-domain knowledge

    • Necessary for scalability in all areas

      • Security

      • Configuration

      • Fault tolerance & recovery

21: xrootd


Scalable proxy security

2

3

2

1

Scalable Proxy Security

SLAC PROXY OLBD

RAL PROXY OLBD

Data Servers

Data Servers

Firewall

1 Authenticate & develop session key

2 Distribute session key to authenticated subscribers

3 Data servers can log into each other using session key

22: xrootd


Proxy performance
Proxy Performance

  • Introduces minimal latency overhead

    • Virtually undetectably from US/Europe

    • Negligible on faster links

      • 2% slower on fast US/US links

      • 10% slower on LAN

  • Can be further improved

    • Parallel streams

    • Better window size calculation

    • Asynchronous I/O

23: xrootd


Conclusion
Conclusion

  • xrootd provides high performance file access

    • Unique performance, usability, scalability, security, compatibility, and recoverability characteristics

  • Should scale to tens of thousand clients

    • Can support tens of thousand of servers

  • Distributed as part of the CERN root package

    • Open software, supported by

      • SLAC (server) and INFN-Padova (client)

24: xrootd


ad