The File System in Linux

1. The File System in Linux • The Linux file system is organized in a tree structure. The files on this tree are one of the following types: • An ordinary file • A directory file • A special file (representing an I/O device) • A link that points to another file • A socket that is used for interprocess communication

2. Directories • A directory is a file whose content consists of directory entries for the files in the directory. • A directory entry contains the name of the file and a pointer to the file. The pointer is an index called the i-number to a table known as the i-list. • Each entry in the i-list is an i-node containing status and address information about a file or is free. • The entire file system may contain several self-contained filesystems, each with its own i-list.

3. Directories grail:/users/faculty/arndt> df /home0 (arthur.cba:/home0 ): 14736728 blocks -1 i-nodes /home1 (arthur.cba:/home1 ): 7846106 blocks -1 i-nodes /userspace (shamu.cba:/export/home0/userspace): 1327190 blocks -1 i-nodes /usr (/dev/vg00/lvol3 ): 65700 blocks 13437 i-nodes /tmp (/dev/vg00/lvol4 ): 219084 blocks 12016 i-nodes /users (/dev/vg00/lvol5 ): 245610 blocks 65662 i-nodes /mnt1 (/dev/vg00/lvol6 ): 129708 blocks 17093 i-nodes /client_server (/dev/dsk/c4d0s2 ): 512710 blocks 357634 i-nodes / (/dev/vg00/lvol1 ): 23880 blocks 6769 i-nodes grail:/users/faculty/arndt>

4. Special Files • Linux represents I/O devices such as terminals, printers, tape, and disk drives as special files in the file system. • In this way, an application program can treat file and device I/O in the same way. • Each special file is stored in the directory /dev. • A special file can be either a character special file or a block special file. • A character special file represents a character-oriented I/O device. • A block special represents a high-speed I/O device that transfers data in blocks rather than bytes.

5. Links • A directory entry may be a pointer to another file. This is called a link. There are two kinds of links: • hard links • symbolic links • A hard link is an entry in a directory with a name and some other file’s i-number. The hard link is not distinguishable from the original file. • A file may have several links to it. The link must be to an ordinary file in the same filesystem.

6. Links • A link is created to another file by using the ln command: ln file ln file linkname ln file1 . . . Dir • By default, a hard link is created by ln. • It is not necessary to be the owner of a file to link to the file. The command rm removes the directory entry of the file. The file is physically deleted when the last link to it is removed. • The number of hard links to a file is kept as part of the status.

7. Symbolic Links • A symbolic link is a directory entry that contains the pathname of another file. The symbolic link can be used as an argument in a command. • The file pointed to can be removed and the link will remain. • A symbolic link can span filesystems and may point to a directory. • A symbolic link can be created using ln -s

8. File Access Control • The file mode for a file as displayed by ls -l is, for example: -rw-rw-rw- • The first position shows the file type: - for normal, d for directory, l for symbolic link, c for character special file, b for block special file. • Positions 2-4 show read, write, and execute permission for the owner of the file. A - means no permission.

9. File Access Control • Positions 5-7 show the permissions for users in the file group. • Positions 8-10 show the permissions for other users. • To access a directory, you must have execute permission for that directory. To list its contents, you must have read permission. Write permission is necessary to create or delete a file in a directory • New files are created with a default protection. To change this default, use the command umask. This command uses three octal numbers.

10. File Status • A file status is kept in the i-node for each file in the Linux file system. The file status include: • mode • number of links • owner, group • size • last access, last content change, last status change • i-number • device • block size • block count

11. File Mode • The file mode consists of 16 bits. We have already seen the low order 9 bits: the permissions. The four high bits specify the file type while the next three bits define the way in which an executable file is run. • Set userid: owner of a file changed to user during execution. • Set groupid: sets group id during execution. • The stickybit keeps a shared binary in memory. • The mode of a file may be changed using the chmod command

12. File Mode grail:/usr/bin> ls -l e* | more -r-xr-xr-t 6 bin bin 245760 Sep 7 1993 e -r-xr-xr-t 6 bin bin 245760 Sep 7 1993 edit -rwxr-sr-x 1 root bin 479232 Jun 17 1996 elm -r-xr-sr-x 1 bin mail 270336 Sep 7 1993 elm.orig -r-xr-xr-x 1 bin bin 32768 Jun 17 1996 elmalias -r-sr-xr-x 1 lp bin 20480 Oct 22 1993 enable -r-xr-xr-t 6 bin bin 245760 Sep 7 1993 ex -r-xr-xr-x 1 bin bin 16384 Sep 7 1993 expand grail:/usr/bin> Note the ‘s’ and ‘t’ above in place of ‘x’.

13. Groups • A user can belong to one or more groups specified in the file /etc/group. • Use the command ‘groups’ to see which groups you belong to. • Each file is associated with a group owner. To change the group owner of a file use ‘chgrp’ • This allows groups to work on the same files (but it doesn’t coordinate their changes!).

14. File System Implementation • A Linux file is an array of bytes stored in a number of data blocks in a specific filesystem. • Originally, UNIX blocks were 512 bytes. Some implementations now use 4K bytes per block. Since the final block may waste space if it is not full, it may be broken up into fragments of (e.g.) 1K bytes. • As said previously, information on files is stored in an i-node, where the i-nodes for a filesystem are stored in an i-list.

15. File System Implementation • The i-node also contains pointers to the data blocks of the file. An i-node contains 10 pointers to data blocks, a pointer to a data block containing pointers to data blocks, and a pointer to a data block containing pointers to data blocks containing pointers to data blocks (!). This allows for extremely large files. • Why use this arrangement rather than just a chain of data blocks?

16. Removable Filesystems • One of the Linux filesystems is the root filesystems. All of the other filesystems are removable. • Removable filesystems are attached to the root filesystem at some leaf node using the mount command: mount devfile directory [-r] • The command umount is used to unmount a removable filesystem. Mount as read only

17. Super Block and Cylinder Groups • The super block of a filesystem records information about the filesystem: • Size of the filesystem • Block/fragment size • Length of the i-node list • A list of free blocks and length of the list • A list of free i-nodes and length of the list • The super block is stored at the beginning of a filesystem.

18. Super Block and Cylinder Groups • Linux systems use an improved organization based on the idea of a cylinder group (a set of physically close cylinders on the disk). • I-nodes are distributed within each cylinder group rather than at the beginning of the filesystem. Therefore, i-nodes are close to their data blocks, and access time is improved. Super blocks are also duplicated in order to increase robustness.

19. Filesystem Table • Each filesystem is represented by its own block-type special file. The name of this file along with other information is kept in /etc/checklist or /etc/fstab: /dev/vg00/lvol1 / hfs defaults 0 1 31484 564 /dev/vg00/lvol2 swap ignore sw 0 0 16392 592 /dev/vg00/lvol3 /usr hfs defaults 0 2 0 /dev/vg00/lvol4 /tmp hfs defaults 0 2 13 16408 /dev/vg00/lvol5 /users hfs quota,defaults 0 2 0 /dev/vg00/lvol6 /mnt1 hfs rw,suid 0 3 0 /dev/dsk/c4d0s2 /client_server hfs quota,defaults 0 2 0 arthur.cba:/home0 /home0 nfs rw 0 2 0 arthur.cba:/home1 /home1 nfs rw 0 2 0 shamu.cba:/export/home0/userspace /userspace nfs rw 0 2 0

20. Filesystem Table • The currently mounted tables are kept in /etc/mtab: /dev/vg00/lvol1 / hfs defaults 0 1 917804700 1 /dev/dsk/c4d0s2 /client_server hfs quota 0 2 917804709 1 /dev/vg00/lvol6 /mnt1 hfs defaults 0 3 917804709 1 /dev/vg00/lvol5 /users hfs quota 0 2 917804709 1 /dev/vg00/lvol4 /tmp hfs defaults 0 2 917804709 1 /dev/vg00/lvol3 /usr hfs defaults 0 2 917804709 1 shamu.cba:/export/home0/userspace /userspace nfs rw 0 2 917804764 0 arthur.cba:/home1 /home1 nfs rw 0 2 917804764 0 arthur.cba:/home0 /home0 nfs rw 0 2 917804764 0

21. Network File System • Linux systems allow file operations on remote filesystems stored on other computers on a LAN. This is done using NFS. • A machine on a LAN can function as file server (e.g. by having all users home directories on that machine). • NFS can work on different hardware and OSs (e.g. different varieties of UNIX). • A filesystem is made available remotely by exporting it. It can then be mounted on a remote machine.

22. File System Consistency • In order to improve filesystem performance, a pool of data blocks are kept in memory so that further references to them don’t require a disk access. • Changes made to these blocks are not immediately reflected on disk. • Periodically, they are written to disk. • This arrangement is known as lazy write. • DOS used careful write in which changes are written immediately to disk. • Powering down the computer without going through a shutdown sequence can cause unwritten data to be lost.

23. File System Consistency • The program fsck can be run when the system is rebooted to check for and repair file system inconsistency (compare Windows chkdisk program). • This loss of data is unacceptable for mission-critical database management systems. • Therefore these systems completely bypass the Linux filesystem, managing a raw device by themselves in order to ensure a consistent state of the database in case of system failure.

24. Problems • The user types in the command “cat /user/home/me/sub/file1” - what are the steps involved in retrieving the data in the file? • A removable hard drive formatted for Linux (i.e. Firewire or USB) is attached to a Linux system. What steps are taken in order to make the data on the drive available to application programs?

25. Windows Filesystems • Windows can use multiple filesystem types: • FAT (for compatibility with non-NT systems) • HPFS (the native OS/2 file system) • NTFS (the native NT file system type) • NTFS incorporates the normal features of other modern filesystems. • Journaling • Enhanced file security • Lazy-write scheme • Mirrored master file table and boot sector

26. Journaling • Like AIX, Windows uses techniques borrowed from real-time transaction processing to ensure file system structure integrity at all times. • Before any change is made to the filesystem, it is first recorded in a designated log file. • During a system boot, the filesystem structure is automatically made current with its log. • This doesn’t prevent data loss since only changes to the structure of the filesystem are logged, not changes to a file’s contents.

27. Lazy-write scheme • Like Linux, disk operations are performed to a cache in memory rather than directly to disk. • This optimizes system I/O performance, but may result in file changes being lost in case of a power failure. • Windows also uses the lazy-write scheme on FAT filesystems, unlike DOS. • The journaling technique allow for a quicker recovery than other lazy-write systems - chkdsk is not required to scan the disk in case of crash.

28. Master File Table • The equivalent of the i-list in Windows is the Master File Table (MFT) • The first 16 records in the MFT contains metadata files used to implement the file system: • Master File Table: a list of all contents of the NTFS volume • MFT2: a mirror of the first three records of the MFT, used to guarantee access to the MFT in case of a single-sector error • Log File: a list of transaction steps used for recoverability. • Volume: the volume name, NTFS version, and other info. • Attribute Def. Table: A table of attribute names, numbers, and descriptions

29. File System Consistency • Root Filename Index: Root folder • Cluster Bitmap: A representation of the clusters showing which clusters (i.e. blocks) are in use. • Partition Boot Sector: the bootstrap for the volume, if bootable. • Bad Cluster File: List of bad clusters. • Quota table: Disk quota usage for each user on a volume. Currently unused. • Upcase Table: Used for converting lowercase characters to the matching Unicode uppercase characters. • The next 5 MFT records are reserved for future use.

30. MFT Records • A file in an NTFS filesystem is a collection of attributes stored in an MFT record (and possibly elsewhere on the disk). Attributes include the following: • Basic file information, such as the file’s DOS attributes, modification and access time, and number of hard links. • Filename: Windows filenames may be up to 255 characters in length and may contain any letters, number or Unicode characters. A DOS-style filename may also be given. • Security attributes: the Windows access permissions. • Data attribute: the data in the file. • Additional indexing data for directories.

31. Advanced Filesystem Features • NTFS has optional transparent file compression. • On Windows, all local filesystems are automatically mounted when the the system boots. • Filesystem volumes (as defined with a unique drive letter) can occupy more than one disk partition. • Striped filesystems may be created for higher I/O performance.

32. Advanced Filesystem Features • RAID filesystems are also available. • RAID (Redundant Array of Inexpensive Disks) is a set of six specific disk organizations that can be used to improve disk performance, or reliability, or both. • Most RAIDs consist of a RAID SCSI controller plus a box of SCSI disks that appear to the OS as a single large disk. No software changes are required to use the RAID.

33. RAID • The data in a RAID are distributed over the drives, to allow parallel operation. Several schemes were devised by Patterson and are known as RAID level 0 through RAID level 5. • RAID level 0 stripes data over multiple drives in round robin fashion. In this way, a block of data can be read from multiple drives, working in parallel. This works best when data is requested in large chunks. The controller must coordinate the request and issue the individual requests to the disks.

34. RAID • RAID level 0 is less robust than using a single disk, since the failure of any disk leads to failure for the RAID. • RAID level 1 is a true RAID. It duplicates all the disks so that there are four primary and four backup disks. On a write, a strip is written twice. On a read, either copy can be read. Fault tolerance is excellent. Level 1 RAID is supported on NT Server only.

35. RAID • RAID levels 4 and 5 work with strips of data, with parity. RAID level 4 writes a parity strip (computed from the strips on each of the other disks) on an extra drive. If a drive crashes, the lost bytes can be recomputed from the parity drive. • RAID level 5 distributes the parity bits uniformly over all the disks so that the parity drive does not become a bottleneck. This makes recovery more complex, however. Once again, NT Server only.

37. Removable Disks • FAT and NTFS filesystems may be created on removable disks in Zip, Jaz, floppy disks, etc. • Such disks are removable in the normal manner when the FAT filesystem is used. • For disks formatted with the NTFS system, the disks can be changed only at boot-time.

38. Network Filesystem Access • Files and directories can be shared across a network using three mechanisms: • Windows shares: the native facility for sharing disk resources within a Microsoft network. • The Distributed File System: similar to NFS, allows disk resources of different Windows NT computers to be combined into a single, network-accessible directory tree. • NFS and SAMBA: third party resources allow the sharing of filesystems resources between Windows and UNIX systems.

39. File Types • There are several different Windows file types: • Regular files • Directories • Hard links • Shortcuts (similar to symbolic links, but used in the GUI environment) • Note that there are fewer file types than in Linux since I/O devices, sockets are implemented differently.

40. File Ownership and Permissions • FAT filesystems offer virtually nothing in the way of file and directory access controls. NTFS on the other hand, offers sophisticated controls. • Every file in an NTFS filesystem has an owner associated with it. There is no group owner. • Access control lists (ACLs) are used to grant and limit access to files and directories. • ACLs are composed of name-permission pairs known as access control entries (ACEs).

41. File Ownership and Permissions • Windows uses the following basic permissions: Read (R), Write (W), Execute (X), Delete (D), Change Permissions (P), Take Ownership (O). There are also named commonly used subsets of these permissions. • It is easier to assign permissions to groups, rather than individual users. • ACLs can be viewed using the security tab of the file’s Properties dialog box.