File System Implementation

File System Implementation CISC3595, Fall 09

Outline • File system introduction • File system implementation • Disk space allocation and management • Efficiency and Performance • Recovery • NFS

Objectives for a File Management System • Meet the data management needs of the user • Provide I/O support for a variety of storage device types • Provide a standardized set of I/O interface routines to user processes • Provide I/O support for multiple users (if needed) • Guarantee that the data in the file are valid • Minimize lost or destroyed data • Optimize performance

Requirements for a general purpose system • user should be able to create, delete, read, write and modify files • user may have controlled access to other users’ files • user may control what type of accesses are allowed to his/her files • user should be able to restructure his/her files • user should be able to move data between files • user should be able to back up and recover files in case of damage • user should be able to access files using symbolic names

File-System Structure • File: logical storage unit, collection of related information • File system resides on secondary storage (disks), or tertiary storage • File system organized into layers

File-System Structure • Logical file system: manage metadata information, all file system structures except actual data (i.e., file content) • directory structure, file-control block, protection and security • File-Organization module: • File allocation: map logic block addr.(2nd block of a file) to physical block addr. • Free space management • Basic file system • Issue commands to device driver • Manage memory buffers and caches • Buffer allocated before data transfer • Cache for storing meta-data

File-System Structure • I/O control: device drivers and interrupt handlers • Responsible for starting I/O operations on a device • Processes completion of an I/O request • Translate high-level requests to low-level, hardware-specific instructions (sent to disk controller)

Virtual File Systems • Virtual File Systems (VFS): • same system call interface (API) used for different types of concrete file systems • Support numerous file system types • ext2, ufs, fat, vfat, hpfs, minix, isofs, sysv, hfs, affs, NTFS • NFS, CoDA, AFS ncpfs • Procfs, umsdos, userfs

Mount • Various file systems are mounted at different directories (mounting points) in the files system name space $ mount $ /dev/sda3 on / type ext3 (rw,relatime,errors=remount-ro) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) varrun on /var/run type tmpfs (rw,nosuid,mode=0755) varlock on /var/lock type tmpfs (rw,noexec,nosuid,nodev,mode=1777) udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) fusectl on /sys/fs/fuse/connections type fusectl (rw) lrm on /lib/modules/2.6.28-11-generic/volatile type tmpfs (rw,mode=755) securityfs on /sys/kernel/security type securityfs (rw) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev) gvfs-fuse-daemon on /home/zhang/.gvfs type fuse.gvfs-fuse-daemon (rw,nosuid,nodev,user=zhang)

File system structures (metadata) on disk • Boot control block: contain info. needed for boot an OS from the volume • Volume control block • contains volume (partition) detials: number of blocks, size of blocks, free-block count and list, free FCB count and FCB pointers • Directory structure (per file system): organize files • Per-file FCB: information about a file

File system structures in memory • Mount table: contains info. about each mounted volume • In-memory directory-structure cache: contains recent accessed directory info. • For a directory that is a mounting point, contains flag indicating it’s a mount point, and a pointer to an entry in mount table • System-wide open-file table: a copy of PCB for each open file • Per-process open-file table: contains pointer to appropriate entry in system-wide open-file table • Buffer: hold file system blocks being read from disk or written to disk

Supporting file system interface: file creation • Program issues system call, open(), create(), fopen(), … • Logical files system • allocates a new FCB • reads directory into memory, update it with new file and its FCB, write it back • Call file-organization module to map directory I/O to disk-block number directory I/O disk block #

Supporting file system interface: open a file • Program issues system call, open(), passing a file name • Logic file system (handler of open()) searches system-wide open-file table for the file, if not found, search directory structure for the file, cache directory info, copy file’s PCB into system-wide open-file table • In per-process open-file table, creates an entry for the file, to store pointer to system-wide open-file table entry, current location pointer, access mode info. • Return a pointer to the per-process open-file table, i.e., file descriptor in Unix, or file handler in Windows

Open/Read a file

Directory: • Contains information about files • File Name • File type • File Organisation • For systems that support different organizations • Attributes, ownership • Location: • Volume: Indicates device on which file is stored • Starting Address • Size Used : Current size of the file in bytes, words, or blocks • Size Allocated : The maximum size of the file

Operations Performed on a Directory • A directory system should support a number of operations including: • Search • Create files • Deleting files • Listing directory • Updating directory

Directory Implementation • Linear list of file names with pointer to the data blocks. • simple to program • time-consuming to search • Sorted list? Tree structure? • Hash Table – linear list with a hash table • hash table takes a value computed from file name and returns a pointer to the file name in a linear list • decreases directory search time • collisions – situations where two file names hash to the same location

Allocation Methods • Allocate disk space to files • Contiguous allocation • Linked allocation • Indexed allocation

Contiguous Allocation • Each file occupies a set of contiguous blocks on the disk • Pros: • Simple – only starting location (block #) and length (number of blocks) are required • Random access • Cons: • Wasteful of space: dynamic storage-allocation problem: how to satisfy request from list of non-contiguous free holes • External fragmentation • Files cannot grow

Linked Allocation • Each file is a linked list of disk blocks: blocks may be scattered anywhere on the disk. • Directory contains pointer to the first and last blocks of the file. • Pros: • Solve problems with contiguous allocation • Cons: • Inefficient direct access • Reliability issues

File-Allocation Table • File-allocation table (FAT) – disk-space allocation used by MS-DOS and OS/2. • FAT: for each volume • one entry for each disk block • Pros? • Cons?

Indexed Allocation • Brings all pointers (block #s) together into index block. • Logical view. • Each file has its own index block index table

Indexed Allocation

Indexed Allocation: Multi-level indexing  outer-index file index table

Combined Scheme: UNIX inode (4K bytes per block)

Free-Space Management • Bit vector (n blocks) • First free Block number: • (number of bits per word) *(number of 0-value words) + offset of first 1 bit • Easy to get contiguous free blocks • Bit map requires extra space • E.g., block size = 212 bytes, disk size = 230 bytes (1 gigabyte) • n = 230/212 = 218 bits (or 32K bytes) 0 1 2 n-1 … 0  block[i] free 1  block[i] occupied bit[i] = 

Free-Space Management (Cont.) • Linked list (free list) • Cannot get contiguous space easily • No waste of space • Grouping • First free block contains address of n free blocks • The n-th block therein contains address of another n free blocks, … • Counting • Free blocks might be contiguous • Keep starting block # and length

Efficiency and Performance • Efficiency dependent on: • disk allocation and directory algorithms • types of data kept in file’s directory entry • Performance • disk cache – separate section of main memory for frequently used blocks • free-behind and read-ahead – techniques to optimize sequential access • improve PC performance by dedicating section of memory as virtual disk, or RAM disk

Page Cache • A page cache caches pages rather than disk blocks using virtual memory techniques • Memory-mapped I/O uses a page cache • Routine I/O through the file system uses the buffer (disk) cache • This leads to the following figure

I/O Without a Unified Buffer Cache

Recovery • Consistency checking – compares data in directory structure with data blocks on disk, and tries to fix inconsistencies • Use system programs to back up data from disk to another storage device (floppy disk, magnetic tape, other magnetic disk, optical) • Recover lost file or disk by restoring data from backup

Log Structured File Systems • Log structured (or journaling) file systems record each update to file system as a transaction • All transactions are written to a log • A transaction is considered committed once it is written to log • However, file system may not yet be updated • Transactions in the log are asynchronously written to file system • When file system is modified, the transaction is removed from log • If file system crashes, all remaining transactions in log must still be performed

The Sun Network File System (NFS) • An implementation and a specification of a software system for accessing remote files across LANs (or WANs) • The implementation is part of the Solaris and SunOS operating systems running on Sun workstations using an unreliable datagram protocol (UDP/IP protocol and Ethernet

NFS (Cont.) • Interconnected workstations viewed as a set of independent machines with independent file systems, which allows sharing among these file systems in a transparent manner • A remote directory is mounted over a local file system directory • Mounted directory looks like an integral subtree of local file system, replacing the subtree descending from the local directory • Specification of remote directory for mount operation is nontransparent: host name of remote directory has to be provided • Files in the remote directory can then be accessed in a transparent manner • Subject to access-rights accreditation, potentially any file system (or directory within a file system), can be mounted remotely on top of any local directory

NFS (Cont.) • NFS is designed to operate in a heterogeneous environment of different machines, operating systems, and network architectures; the NFS specifications independent of these media • This independence is achieved through the use of RPC primitives built on top of an External Data Representation (XDR) protocol used between two implementation-independent interfaces • NFS specification distinguishes between the services provided by a mount mechanism and the actual remote-file-access services

NFS Protocol • Provides a set of remote procedure calls for remote file operations. The procedures support the following operations: • searching for a file within a directory • reading a set of directory entries • manipulating links and directories • accessing file attributes • reading and writing files • NFS servers are stateless; each request has to provide a full set of arguments (NFS V4 is just coming available – very different, stateful) • Modified data must be committed to the server’s disk before results are returned to the client (lose advantages of caching) • The NFS protocol does not provide concurrency-control mechanisms

Three Major Layers of NFS Architecture • UNIX file-system interface (based on the open, read, write, and close calls, and file descriptors) • Virtual File System (VFS) layer – distinguishes local files from remote ones, and local files are further distinguished according to their file-system types • The VFS activates file-system-specific operations to handle local requests according to their file-system types • Calls the NFS protocol procedures for remote requests • NFS service layer – bottom layer of the architecture • Implements the NFS protocol

Schematic View of NFS Architecture

NFS Path-Name Translation • Performed by breaking the path into component names and performing a separate NFS lookup call for every pair of component name and directory vnode • To make lookup faster, a directory name lookup cache on the client’s side holds the vnodes for remote directory names

NFS Remote Operations • Nearly one-to-one correspondence between regular UNIX system calls and the NFS protocol RPCs (except opening and closing files) • NFS adheres to the remote-service paradigm, but employs buffering and caching techniques for the sake of performance • File-blocks cache – when a file is opened, the kernel checks with the remote server whether to fetch or revalidate the cached attributes • Cached file blocks are used only if the corresponding cached attributes are up to date • File-attribute cache – the attribute cache is updated whenever new attributes arrive from the server • Clients do not free delayed-write blocks until the server confirms that the data have been written to disk

File System Implementation