Outline for Today’s Lecture

1. Outline for Today’s Lecture Administrative: • Midterm questions? Objective: • Beginning of I/O and File Systems

2. File System Issues • What is the role of files? What is the file abstraction? • File naming. How to find the file we want?Sharing files. Controlling access to files. • Performance issues - how to deal with the bottleneck of disks? What is the “right” way to optimize file access?

3. Role of Files • Persistence - long-lived - data for posterity • non-volatile storage media • semantically meaningful (memorable) names What are the challenges in delivering this functionality?

4. Abstractions User view Addressbook, record for Duke CPS Application addrfile ->fid, byte range* fid File System bytes block# device, block # Disk Subsystem surface, cylinder, sector

5. *File Abstractions • UNIX-like files • Sequence of bytes • Operations: open (create), close, read, write, seek • Memory mapped files • Sequence of bytes • Mapped into address space • Page fault mechanism does data transfer • Named, Possibly typed

6. User grp others rwx rwx rwx 111 100 000 O_RDONLYO_WRONLY O_RDWR O_CREAT O_APPEND ... Relative to beginning, current position, end of file Unix File Syscalls int fd, num, success, bufsize; char data[bufsize]; long offset, pos; fd = open (filename, mode [,permissions]); success = close (fd); pos = lseek (fd, offset, mode); num = read (fd, data, bufsize); num = write (fd, data, bufsize);

7. UNIX File System Calls Open files are named to by an integer file descriptor. Pathnames may be relative to process current directory. char buf[BUFSIZE]; int fd; if ((fd = open(“../zot”, O_TRUNC | O_RDWR) == -1) { perror(“open failed”); exit(1); } while(read(0, buf, BUFSIZE)) { if (write(fd, buf, BUFSIZE) != BUFSIZE) { perror(“write failed”); exit(1); } } Process passes status back to parent on exit, to report success/failure. Process does not specify current file offset: the system remembers it. Standard descriptors (0, 1, 2) for input, output, error messages (stdin, stdout, stderr).

8. R, W, X, none Shared, Private, Fixed, Noreserve Memory Mapped Files fd = open (somefile, consistent_mode); pa = mmap(addr, len, prot, flags, fd, offset); fd + offset pa len len VAS Reading performed by Load instr.

9. Functions of Device Subsystem In general, deal with device characteristics • Translate block numbers (the abstraction of device shown to file system) to physical disk addresses. Device specific (subject to change with upgrades in technology) intelligent placement of blocks. • Schedule (reorder?) disk operations

10. Disk Devices

11. What to do about Disks? • Disk scheduling • Idea is to reorder outstanding requests to minimize seeks. • Layout on disk • Placement to minimize disk overhead • Build a better disk (or substitute) • Example: RAID

12. Avoiding the Disk -- Caching

13. File Buffer Cache Proc • Avoid the disk for as many file operations as possible. • Cache acts as a filter for the requests seen by the disk - reads served best. • Delayed writeback will avoid going to disk at all for temp files. Memory File cache

14. Handling Updates in the File Cache 1. Blocks may be modified in memory once they have been brought into the cache. Modified blocks are dirty and must (eventually) be written back. 2. Once a block is modified in memory, the write back to disk may not be immediate (synchronous). Delayed writes absorb many small updates with one disk write. How long should the system hold dirty data in memory? Asynchronous writes allow overlapping of computation and disk update activity (write-behind). Do the write call for block n+1 while transfer of block n is in progress.

15. Linux Page Cache • Page Cache is the disk cache for all page-based I/O – subsumes file buffer cache. • All page I/O flows through page cache • pdflush daemons – writeback to disk any dirty pages/buffers. • When free memory falls below threshold, wakeup daemon to reclaim free memory • Specified number written back • Free memory above threshold • Periodically, to prevent old data not getting written back, wakeup on timer expiration • Writes all pages older than specified limit.

16. Disk Scheduling – Seek Opt.

17. Rotational Media Track Sector Arm Cylinder Platter Head • Access time = seek time + rotational delay + transfer time • seek time = 5-15 milliseconds to move the disk arm and settle on a cylinder • rotational delay = 8 milliseconds for full rotation at 7200 RPM: average delay = 4 ms • transfer time = 1 millisecond for an 8KB block at 8 MB/s

18. Disk Scheduling • Assuming there are sufficient outstanding requests in request queue • Focus is on seek time - minimizing physical movement of head. • Simple model of seek performance Seek Time = startup time (e.g. 3.0 ms) + N (number of cylinders ) * per-cylinder move (e.g. .04 ms/cyl)

19. “Textbook” Policies 1, 3, 2, 4, 3, 5, 0 • Generally use FCFS as baseline for comparison • Shortest Seek First (SSTF) -closest • danger of starvation • Elevator (SCAN) - sweep in one direction, turn around when no requests beyond • handle case of constant arrivals at same position • C-SCAN - sweep in only one direction, return to 0 • less variation in response FCFS SSTF SCAN CSCAN

20. Sector Scheduling

21. Linux Disk Schedulers • Linus Elevator • Merging and sorting: when new request comes in • Merge with any enqueued request for adjacent sector • If any request is too old, put new request at end of queue • Sort by sector location in queue (between existing requests) • Otherwise at end • Deadline – each request placed on 2 of 3 queues • sector-wise – as above • read FIFO and write FIFO – whenever expiration time exceeded, service from here • Anticipatory • Hang around waiting for subsequent request just a bit

22. Disk Layout

23. Layout on Disk • Can address both seek and rotational latency • Cluster related things together (e.g. an inode and its data, inodes in same directory (ls command), data blocks of multi-block file, files in same directory) • Sub-block allocation to reduce fragmentation for small files • Log-Structure File Systems

24. The Problem of Disk Layout • The level of indirection in the file block maps allows flexibility in file layout. • “File system design is 99% block allocation.” [McVoy] • Competing goals for block allocation: • allocationcost • bandwidth for high-volume transfers • efficient directory operations • Goal: reduce disk arm movement and seek overhead. • metric of merit: bandwidth utilization

25. Data Block Addr ... File Attributes ... ... ... ... ... UNIX Inodes 3 3 3 3 Data blocks Block Addr 1 2 2 ... Decoupling meta-data from directory entries 1 2 2 1

26. FFS Cylinder Groups • FFS defines cylinder groups as the unit of disk locality, and it factors locality into allocation choices. • typical: thousands of cylinders, dozens of groups • Strategy: place “related” data blocks in the same cylinder group whenever possible. • seek latency is proportional to seek distance • Smear large files across groups: • Place a run of contiguous blocks in each group. • Reserve inode blocks in each cylinder group. • This allows inodes to be allocated close to their directory entries and close to their data blocks (for small files).

27. FFS Allocation Policies 1. Allocate file inodes close to their containing directories. For mkdir, select a cylinder group with a more-than-average number of free inodes. For creat, place inode in the same group as the parent. 2. Concentrate related file data blocks in cylinder groups. Most files are read and written sequentially. Place initial blocks of a file in the same group as its inode. How should we handle directory blocks? Place adjacent logical blocks in the same cylinder group. Logical block n+1 goes in the same group as block n. Switch to a different group for each indirect block.

28. Allocating a Block 1. Try to allocate the rotationally optimal physical block after the previous logical block in the file. Skip rotdelay physical blocks between each logical block. (rotdelay is 0 on track-caching disk controllers.) 2. If not available, find another block a nearby rotational position in the same cylinder group We’ll need a short seek, but we won’t wait for the rotation. If not available, pick any other block in the cylinder group. 3. If the cylinder group is full, or we’re crossing to a new indirect block, go find a new cylinder group. Pick a block at the beginning of a run of free blocks.

29. Clustering in FFS • Clustering improves bandwidth utilization for large files read and written sequentially. • Allocate clumps/clusters/runs of blocks contiguously; read/write the entire clump in one operation with at most one seek. • Typical cluster sizes: 32KB to 128KB. • FFS can allocate contiguous runs of blocks “most of the time” on disks with sufficient free space. • This (usually) occurs as a side effect of setting rotdelay = 0. • Newer versions may relocate to clusters of contiguous storage if the initial allocation did not succeed in placing them well. • Must modify buffer cache to group buffers together and read/write in contiguous clusters.

30. Effect of Clustering • Access time = seek time + rotational delay + transfer time • average seek time = 2 ms for an intra-cylinder group seek, let’s say • rotational delay = 8 milliseconds for full rotation at 7200 RPM: average delay = 4 ms • transfer time = 1 millisecond for an 8KB block at 8 MB/s 8 KB blocks deliver about 15% of disk bandwidth. 64KB blocks/clusters deliver about 50% of disk bandwidth. 128KB blocks/clusters deliver about 70% of disk bandwidth. • Actual performance will likely be better with good disk layout, since most seek/rotate delays to read the next block/cluster will be “better than average”.

31. Disk Alternatives

32. Build a Better Disk? • “Better” has typically meant density to disk manufacturers - bigger disks are better. • I/O Bottleneck - a speed disparity caused by processors getting faster more quickly • One idea is to use parallelism of multiple disks • Striping data across disks • Reliability issues - introduce redundancy

33. RAID Redundant Array of Inexpensive Disks Striped Data Parity Disk (RAID Levels 2 and 3)

34. MEMS-based StorageGriffin, Schlosser, Ganger, Nagle • Paper in OSDI 2000 on OS Management • Comparing MEMS-based storage with disks • Request scheduling • Data layout • Fault tolerance • Power management

35. Settling time after X seek • Spring factor - non-uniform over sled positions • Turnaround time

36. Data on Media Sled

37. Disk Analogy • 16 tips • MxN = 3 x 280 • Cylinder – same x offset • 4 tracks of 1080 bits, 4 tips • Each track – 12 sectors of 80 bits (8 encoded bytes) • Logical blocks striped across 2 sectors

38. Logical Blocks and LBN • Sectors are smaller than disk • Multiple sectors can be accessed concurrently • Bidirectional access

39. MEMS Positioning – X and Y seek (0.2-0.8 ms) Settling time 0.2ms Seeks near edges take longer due to springs, turnarounds depend on direction – it isn’t just distance to be moved. More parts to break Access parallelism Disk Seek (1-15 ms) and rotational delay Settling time 0.5ms Seek times are relatively constant functions of distance Constant velocity rotation occurring regardless of accesses Comparison

40. File System

41. Functions of File System • (Directory subsystem) Map filenames to fileids-open (create) syscall. Create kernel data structures.Maintain naming structure (unlink, mkdir, rmdir) • Determine layout of files and metadata on disk in terms of blocks. Disk block allocation. Bad blocks. • Handle read and write system calls • Initiate I/O operations for movement of blocks to/from disk. • Maintain buffer cache

42. r-w pos, mode r-w pos, mode pos pos File System Data Structures System-wide Open file table System-wide File descriptor table Process descriptor in-memory copy of inode ptr to on-disk inode File data stdin stdout per-process file ptr array stderr

43. Data Block Addr ... File Attributes ... ... ... ... ... UNIX Inodes 3 3 3 3 Data blocks Block Addr 1 2 2 ... Decoupling meta-data from directory entries 1 2 2 1

44. File Sharing Between Parent/Child main(int argc, char *argv[]) { char c; int fdrd, fdwt, fdpriv; if ((fdrd = open(argv[1], O_RDONLY)) == -1) exit(1); if ((fdwt = creat([argv[2], 0666)) == -1) exit(1); fork(); if ((fdpriv = open([argv[3], O_RDONLY)) == -1) exit(1); while (TRUE) { if (read(fdrd, &c, 1) != 1) exit(0); write(fdwt, &c, 1); } }

45. r-w pos, mode r-w pos, mode forked process’s Process descriptor openafterfork File System Data Structures System-wide Open file table System-wide File descriptor table Process descriptor in-memory copy of inode ptr to on-disk inode stdin stdout per-process file ptr array stderr

46. user ID process ID process group ID parent PID signal state siblings children user ID process ID process group ID parent PID signal state siblings children Sharing Open File Instances shared seek offset in shared file table entry parent shared file (inode or vnode) child system open file table process file descriptors process objects

47. Goals of File Naming • Foremost function - to find files (e.g., in open() ), Map file name to file object. • To store meta-data about files. • To allow users to choose their own file names without undue name conflict problems. • To allow sharing. • Convenience: short names, groupings. • To avoid implementation complications

48. File Attributes File Attributes Directory node Directory node File Attributes current inode# Proj inode# File Attributes Proj Directory node proj3 inode# Pathname Resolution cps110 “cps110/current/Proj/proj3” current proj3 data file index node of wd

49. Linux dcache cps210dentry Inodeobject Hashtable spr04dentry Inodeobject Projdentry Inodeobject Inodeobject proj1dentry

50. Naming Structures • Flat name space - 1 system-wide table, • Unique naming with multiple users is hard.Name conflicts. • Easy sharing, need for protection • Per-user name space • Protection by isolation, no sharing • Easy to avoid name conflicts • Register identifies with directory to use to resolve names, possibility of user-settable (cd)