1 / 43

Distributed File Systems

Distributed File Systems. CSCI 444/544 Operating Systems Fall 2008. Agenda. Distributed file systems File service model Directory service (naming) Stateful vs stateless file servers Caching NFS vs AFS. Client. Client. Client. What is a distributed file system?. File Servers.

claire-lamb
Download Presentation

Distributed File Systems

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Distributed File Systems CSCI 444/544 Operating Systems Fall 2008

  2. Agenda • Distributed file systems • File service model • Directory service (naming) • Stateful vs stateless file servers • Caching • NFS vs AFS

  3. Client Client Client What is a distributed file system? File Servers Network Client Support network-wide sharing of files and disks

  4. Distributed File System • Distributed file system (DFS) – a distributed implementation of the classical time-sharing model of a file system, where multiple users share files and storage resources • A DFS manages set of dispersed storage devices • Overall storage space managed by a DFS is composed of different, remotely located, smaller storage spaces • There is usually a correspondence between constituent storage spaces and sets of files

  5. Benefits • Why are distributed file systems useful? • Access from multiple clients • Same user on different machines can access same files • Simplifies sharing • Different users on different machines can read/write to same files • Simplifies administration • One shared server to maintain (and backup) • Improve reliability • Add RAID storage to server

  6. Challenges • Transparent access • User sees single, global file system regardless of location • Scalable performance • Performance does not degrade as more clients are added • Fault Tolerance • Client and server identify and respond appropriately when other crashes • Consistency • See same directory and file contents on different clients at same time • Security • Secure communication and user authentication • Tension across these goals • Example: Caching helps performance, but hurts consistency

  7. File Service • Two models for file services • upload/download: files move between server and clients, few operations (read file & write file), simple, requires storage at client, good if whole file is accessed • remote access: files stay at server, rich interface with many operations, less space at client, efficient for small accesses

  8. File Service Model Transfer Models (a) upload/download model (b) remote access model (b) (a)

  9. Directory Service • • Provides naming usually within a hierarchical file system • • Clients can have the same view (global root directory) or different views of the file system (remote mounting) • • Naming: mapping between logical and physical objects- Multilevel mapping – abstraction of a file that hides the details of how and where on the disk the file is actually stored • Transparency: a transparent DFS hides the location where in the network the file is stored

  10. Naming Structures • Location transparency – file name does not reveal the file’s physical storage location • The file location is invisible to the user • File name still denotes a specific, although hidden, set of physical disk blocks • Convenient way to share data • Location independence – file name does not need to be changed when the file’s physical storage location changes • Better file abstraction • Promotes sharing the storage space itself • Separates the naming hierarchy form the storage-devices hierarchy

  11. Naming Schemes • Files named by combination of their host name and local name; guarantees a unique systemwide name • Neither location transparent nor location independent • Attach remote directories to local directories, giving the appearance of a coherent directory tree • only previously mounted remote directories can be accessed transparently • Total integration of the component file systems • A single global name structure spans all the files in the system • If a server is unavailable, some arbitrary set of directories on different machines also becomes unavailable

  12. Server System Structure • File + directory service • • Cache directory hints at client to accelerate the path • name look up • – directory and hints must be kept coherent • State information about clients at the server • stateless server: no client information is kept between requests • stateful server: servers maintain state information about clients between requests

  13. Stateful File Server • Mechanism • Client opens a file • Server fetches information about the file from its disk, stores it in its memory, and gives the client a connection identifier unique to the client and the open file • Identifier is used for subsequent accesses until the session ends • Server must reclaim the main-memory space used by clients who are no longer active • Increased performance • Fewer disk accesses • Stateful server knows if a file was opened for sequential access and can thus read ahead the next blocks

  14. Stateless File Server • Avoids state information by making each request self-contained • Each request identifies the file and position in the file • No need to establish and terminate a connection by open and close operations • - run over UDP

  15. Caching • Three possible places: server’s memory, client’s disk, client’s memory • • Caching in server’s memory: avoids disk access but still network access • • Caching at client’s disk (if available): tradeoff between disk access and remote memory access • • Caching at client in main memory - inside each process address space: no sharing at client - in the kernel: kernel involvement on hits - in a separate user-level cache manager: flexible and efficient if paging can be controlled from user-level • Server-side caching eliminates coherence problem. Client-side cache coherence?

  16. Caching

  17. Cache Consistency • Is locally cached copy of the data consistent with the master copy? • Client-initiated approach (poll) • Client initiates a validity check • Server checks whether the local data are consistent with the master copy • Delayed write in NFS • Server-initiated approach (push) • Server records, for each client, the (parts of) files it caches • When server detects a potential inconsistency, it must react • Callback in AFS

  18. Case Study: NFS • Sun’s Network File System • Introduced in 1980s, multiple versions (v2, v3, v4) • has become a common standard for distributed UNIX file access • Key idea #1: Stateless server • Server not required to remember anything (in memory) • Which clients are connected, which files are open, ... • Implication: All client requests have enough info to complete operation (self-contained) • Example: Client specifies offset in file to write to • One Advantage: Server state does not grow with more clients • Key idea #2: Idempotent server operations • Operation can be repeated with same result (no side effects)

  19. Basic Feature • NFS runs over LANs (even over WANs – slowly) • Basic feature • allow a remote directory to be “mounted” (spliced) onto a local directory • Gives access to that remote directory and all its descendants as if they were part of the local hierarchy • Pretty much exactly like a “local mount” or “link” on UNIX • except for implementation and performance …

  20. NFS Overview • NFS is based on the remote access model • • Remote Procedure Calls (RPC) for communication between client • and server • Client Implementation • Provides transparent access to NFS file system • UNIX contains Virtual File system layer (VFS) • Vnode: interface for procedures on an individual file • Translates vnode operations to NFS RPCs • Server Implementation • Stateless: Must not have anything only in memory • Implication: All modified data written to stable storage before return control to client

  21. NSF Protocols • Two client-server protocols: • • The first NFS protocol handles mounting • - Establishes initial logical connection between server and client • • The second NFS protocol is for directory and file access

  22. NFS Mounting Protocol • A client can send a path name to a server and request permission to mount that directory somewhere in its directory hierarchy • If the path name is legal and the directory specified has been exported, the server returns a file handle to the client • Export list – specifies local file systems that server exports for mounting, along with names of client machines that are permitted to mount them • File handle contains fields uniquely identifying the file system type, disk, i-node number of the directory, access right information

  23. 2nd NSF Protocols Second NFS Protocol is for directory and file access Clients send messages to servers to manipulate directories, read and write files Clients access file attributes NFS ‘read’ operation Lookup operation – returns file handle Read operation – uses file handle to read the file Advantage: stateless server open() and close() is intentionally missed 11/1/2014 23

  24. Basic NFS Operations • Operations • lookup(dirfh, name) returns (fh, attributes) • Use mount protocol for root directory • create(dirfh, name, attr) returns (newfs, attr) • remove(dirfs, name) returns (status) • read(fh, offset, count) returns (attr, data) • write(fh, offset, count, data) returns attr • gettattr(fh) returns attr

  25. Three Major Layers of NFS Architecture • UNIX file-system interface (based on the open, read, write, and close calls, and file descriptors) • Virtual File System (VFS) layer – distinguishes local files from remote ones, and local files are further distinguished according to their file-system types • The VFS activates file-system-specific operations to handle local requests according to their file-system types • Calls the NFS protocol procedures for remote requests • NFS service layer – bottom layer of the architecture • Implements the NFS protocol over RPC

  26. Schematic View of NFS Architecture

  27. VFS Virtual file system provides a standard interface, using vnodes as file handles. vnode -- network wide unique (like an inode but for a network) A vnode describes either a local or remote file Path name look up (past mount point) requires RPC per name. Client cache of remote vnodes for remote directory names

  28. Mapping UNIX System Calls to NFS Operations • Unix system call: fd = open(“/dir/foo”) • Traverse pathname to get filehandle for foo • dirfh = lookup(rootdirfh, “dir”); • fh = lookup(dirfh, “foo”); • Record mapping from fd file descriptor to fh NFS filehandle • Set initial file offset to 0 for fd • Return fd file descriptor • Unix system call: read(fd,buffer,bytes) • Get current file offset for fd • Map fd to fh NFS filehandle • Call data = read(fh, offset, bytes) and copy data into buffer • Increment file offset by bytes • Unix system call: close(fd) • Free resources associated with fd

  29. NSF Layer Structure 11/1/2014 29

  30. Client-side Caching • Caching needed to improve performance • Reads: Check local cache before going to server • Writes: Only periodically write-back data to server • Avoid contacting server • Avoid slow communication over network • Server becomes scalability bottleneck with more clients • Two client caches • data blocks • attributes (metadata)

  31. Case Study: AFS • A distributed computing environment (Andrew) under development since 1983 at Carnegie-Mellon University, purchased by IBM and released as Transarc DFS, now open sourced as OpenAFS • AFS tries to solve complex issues such as uniform name space, location-independent file sharing, client-side caching (with cache consistency), secure authentication (via Kerberos) • Also includes server-side caching (via replicas), high availability • Can span 5,000 workstations • Consists of dedicated file servers

  32. AFS Overview • AFS is based on the upload/download model • Opening a file causes it to be cached, in its entirety, on the local disk • Client does as much as possible locally and interact as little as possible with the server • AFS is stateful • The server keeps track of which files are opened by which clients • AFS provides location independence, as well as location transparency • The physical storage location of the file can be changed, without having to change the path of file • Has a single name space • AFS provides callback promise • Inform all clients with open files about any updates made to that file by a client

  33. AFS AFS – Andrew File System workstations grouped into cells for administrative purposes note position of venus and vice /afs • Client's view

  34. File Sharing • AFS enables users to share remote files as easily as local files. To access a file on a remote machine in AFS, you simply specify the file's pathname. • AFS users can see and share all the files under the /afs root directory, given the appropriate privileges. An AFS user who has the necessary privileges can access a file in any AFS cell, simply by specifying the file's pathname. File sharing in AFS is not restricted by geographical distances or operating system differences

  35. Another Client View

  36. ANDREW File Operations • Andrew caches entire files from servers into local disk • A client workstation interacts with Vice servers only during opening and closing of files • Venus – caches files from Vice when they are opened, and stores modified copies of files back when they are closed • Reading and writing bytes of a file are done by the kernel without Venus intervention on the cached copy • Venus caches contents of directories and symbolic links, for path-name translation • Exceptions to the caching policy are modifications to directories that are made directly on the server responsibility for that directory

  37. AFS Naming • Clients are presented with a partitioned space of file names: a local name space and a shared name space • Dedicated servers, called Vice, present the shared name space to the clients as an homogeneous, identical, and location transparent file hierarchy • The local name space is the root file system of a workstation, from which the shared name space descends • Workstations run the Virtue protocol to communicate with Vice, and are required to have local disks where they store their local name space • Servers collectively are responsible for the storage and management of the shared name space

  38. ANDREW Shared Name Space • The storage disks in a computer are divided into sections called partitions. AFS further divides partitions into units called volumes • A fid identifies a Vice file or directory - A fid is 96 bits long and has three equal-length components: • volume number • vnode number – index into an array containing the inodes of files in a single volume • uniquifier – ensures that file identifiers are not reused • Fids are location transparent; therefore, file movements from server to server do not invalidate cached directory contents • Location information is kept on a volume basis, and the information is replicated on each server

  39. AFS caching • Write-on-close: writes are propagated to the server side copy only when the client closes the local copy of the file • The client assumes that its cache is up to date, unless it receives a callback message from the server saying otherwise • on file open, if the client has received a callback on the file, it must fetch a new copy; otherwise it uses its locally-cached copy

  40. ANDREW Implementation • Client processes are interfaced to a UNIX kernel with the usual set of system calls • Venus carries out path-name translation component by component • The UNIX file system is used as a low-level storage system for both servers and clients • The client cache is a local directory on the workstation’s disk • Both Venus and server processes access UNIX files directly by their inodes to avoid the expensive path name-to-inode translation routine

  41. System call interception in AFS

  42. AFS Cache Implementation • Venus manages two separate caches: • one for status • one for data • LRU algorithm used to keep each of them bounded in size • The status cache is kept in virtual memory to allow rapid servicing of stat (file status returning) system calls • The data cache is resident on the local disk, but the UNIX I/O buffering mechanism does some caching of the disk blocks in memory that are transparent to Venus

  43. AFS Commands • AFS commands are grouped into three categories: • File server commands (fs) • - lists AFS server information • - set and list ACLs (access control list) • Protection commands (pts) • - create and manage (ACL) groups • Authentication commands • - klog, unlog, kpasswd, tokens

More Related