Nache: Design and Implementation of a Caching Proxy for NFSv4

1. Nache: Design and Implementation of a Caching Proxy for NFSv4 Ajay Gulati, Rice University Manoj Naik, IBM Almaden Renu Tewari, IBM Almaden

2. Talk Outline • Federated File System • NFSv4: Overview and New Features • Nache Architecture • Nache Implementation • Experimental Results • Conclusions and Discussion

3. Federated File System • File system federation with unified namespace • Flexible data management • Replication, migration and read/write caching across distributed file servers • What are the options? • AFS, DCE/DFS provide distributed FS over WAN, but… • difficult to deploy, not widely used • GPFS, Lustre are cluster filesystems, but… • good for controlled environments, no federation • NFS is a standard, but… • v2/v3 have been designed mainly for LAN, are chatty and not suited for WAN, firewall issues • NFSv4?

4. Why NFSv4? • Open industry standard • Optimizations for WAN • Compound operations • A single RPC request contains multiple ops, reduces round-trips • Read-Write delegations • Efficient cache consistency, no need to contact server once a delegation is obtained • Delegations can be recalled on conflicting access • Federation support • Client redirection • Server returns NFS4ERR_MOVED to redirect clients • Client requests fs_locations attribute • Client connects to new location

5. Unified namespace for clients Admin Filesystem Namespace Unification Almaden Houston New York Clients Separate namespace across servers

6. 5.Client connects to New York server 7. DELEGRETURN 1. OPEN /project/p1 2. NFS4ERR_MOVED 3. GETATTR 4. fs_locations 8. Server responds 6. /project/p1 Data Access using NFSv4 Redirection /project Server (New York) Server (San Jose) Clients (San Jose) Limitations • High network latency when clients access remote servers • Delegation are unsuited for data sharing, fewer awarded and frequently broken • No unified caching, every client has to access remote server

7. What is a Proxy Cache (Nache)? • An NFSv4 server acts as a proxy for another remote NFS server • caches data for reads and writes using delegations • becomes the temporary “owner” of the data servicing opens, lock requests and reads and writes • Multiple clients can share a proxy cache using a single server-to-proxy delegation • Extends beyond web-proxy caching

8. 1.OPEN /project/p1 2. Delegate /project/p1 to proxy 3./project/p1 Data Access using a Proxy Cache Server (New York) Proxy Cache (San Jose) /project Clients (San Jose) Benefits with NFSv4 Proxy • Reduced network traffic and delay • Clients access files through local proxy cache • Clients can share delegations – fewer conflicts/recalls • Improved performance for cached files • Clients can collectively take advantage of aggressive readahead, write back and local locking at the proxy • Both data and delegations are shared

9. Nache Architecture (Linux nfsd module) (Linux nfs module) Nache Server Nache Client Remote Server (mount from) Local Client (export to) VFS Layer cacheFS BufferCache Nache

10. Nache Implementation • Cascaded mounts • NFS operation forwarding • Sub-operation RPC call handling

11. Server Client Nache /net /export /almaden /watson /export mount nache:/ /net mount server:/ /almaden Cascaded Mounts • Proxy exports a NFS mounted file system • Added export functionality for NFS • export_ops added, not available in vanilla NFS ls /net at client shows contents of /watson at server

12. NFS Operation Forwarding • NFS ops are redirected from nfsd to nfs using vfs layer • Issues: certain operations are stateful • OPEN, CREATE, LOCK, ULOCK, CLOSE Application VFS VFS VFS NFSD NFS Client Nache Server Nache Client Server Client Nache

13. 2.LOOKUP 3.ACCESS 4.OPEN (Linux nfsd module) (Linux nfs module) 1.OPEN Nache Server Nache Client Remote Server Local Client VFS Layer Sub-Operation RPC calls • Each FS operation at Nache server translates to a separate RPC call from Nache client • Solution: • Remove such redundant checks from code path 

14. Experimental Methodology • Setup • 2-6 local NFS clients (San Jose) • One local proxy with modified kernel modules (San Jose) • One local NFS server (San Jose) • One remote NFS server (New York) • Benchmarks • Filebench (developed by Sun) • Compilation of various packages – Linux kernel, emacs, gdb • Our micro-benchmarks

15. Experiment Categories • Delegation Benefits • Nache Benefits • Nache Overhead

16. Read Delegation Benefits Client repeats open-read-close on 100 files Server ops reduced by 50% due to delegations

17. Write Delegation Benefits A client repeats open-read/write-close on 100 files Server ops reduced by 4 times

18. Delegation Benefits (Server load) Server ops reduced due to delegations

19. Delegation Benefits (Client performance) Time taken is lower and ops/sec is higher due to delegations

20. Experiment Categories • Delegation Benefits • Nache Benefits • Nache Overhead

21. Nache Benefits - Filebench (webserver) Ops at server reduces by 38% with 4 clients

22. Nache Benefits – Kernel compile Ops at server unaffected by increase in number of clients

23. Nache Benefits – Compile Time Compile time unaffected by increase in number of clients

24. Experiment Categories • Delegation Benefits • Nache Benefits • Nache Overhead

25. Nache Overhead over WAN (Throughput) Nache induces an overhead of 8-43% Code can be further optimized to reduce overheads

26. Nache Overhead over WAN (Latency) Proxy induces an overhead of 8-42% Code can be further optimized to reduce overheads

27. Result Summary • Delegations provide better throughput, lower latency and reduced load at server • Nache improves performance in presence of sharing among clients • Overhead of Nache in absence of sharing is not too high except for few cases and can be further reduced

28. Conclusions • NFSv4 is a good alternative for building federated file systems • Delegations help in reducing the number of server operations and provide better caching • Nache can be integrated with a federated file system architecture and improve performance in presence of sharing