Linux

1. Linux → File Systems Hui Liu, Pengpeng Qi Device Drivers Andrew Henderson

2. File System Hui Liu && Pengpeng Qi

3. What is file system? • File system is a mechanism for organizing data and metadata in a storage device. • Linux looks all the file systems as a general object. • These objects are super block, inode, dentry, and file • as a protocol: another view of file system, just like IP

4. High level architecture • Each implementation of the file system will export a common interface for VFS to use • Linux supports several file systems old : mimix, ext2 new : ext4, JFS • Now we are talking about Reiser4, NTFS, ext4.

5. Reiser4 basic design idea • new features: • Dancing tree • Tail packing • Allocate on flush • Wandering-log • Atomicy • Plugin structure

6. Reiser4features • Dancing tree • Another kind of B*- tree, but Not strictly follow the definition of B*- tree. • Tail packing • A close and continuous storing method, can improve the performance of w/r amount of small files, and enhance the efficiency of the disks. • The Modification and overlay of a file is relatively inefficient • Allocate on flush

7. NTFS • From Microsoft. • Replacing the FAT • Improvement: support metadata, advanced data structure • MFT: stores the information of all the files and directories in the NTFS volumes. Just like a database table. • Clusters: several sectors organized together

8. NTFS • Data in NTFS: all of data in NTFS file system is files • Metadata: contains volume attributes, cluster info, user file list • partition boot sector • Volume Boot Sector • BIOSParameterBlock • VolumeBootCode

9. Ext2/Ext3/Ext4 • Ext2 • Super block and GDT are vital, therefore other groups will use their copies. • Reversed GDT, the “resize feature”, which can expand the size of whole file system. • Ext3 • Extension of ext2, Journaling file system • Ext2 may corrupt after reboot from exception like power reset, journal will ensure file system consistent or recovery file system on system boot.

10. Ext2/Ext3/Ext4 • Limit of ext2 & ext3 • Ext4 • Inherit all the features of ext2/ext3. • Larger file system Max file size: 16TB Max file system size: 1EB

11. Ext4disk layout

12. Ext4 • File system disk layout • Group • Block bitmap • Super block • Group descriptor table • Inode bitmap • Inode table • Features • Backward compatibility • Large file system • Extents • Journal checksumming • No Journaling mode • Persistent preallocation • Delayed allocation • ……

13. Linux File Systems Hui Liu, Pengpeng Qi → Device Drivers Andrew Henderson

14. Basics of Device Driver Modules In The 2.6 Linux Kernel By: Andrew Henderson CIS 657 – Operating Systems

15. Linux Device Driver Types Linux broadly classifies each type of device driver into one of three broad categories: Character device Block device Network interface Other driver classifications exist orthogonal to these: USB device SCSI device

16. Character Devices Similar to what is seen in FreeBSD, character devices access a stream of bytes Some character devices allow for seeking back and forth in a data stream Graphic framebuffers are an example Most character devices are serial data streams Terminals, serial ports, etc.

17. Block Devices Unix block devices typically handle read/write operations in some fixed-size block of memory Block sizes are usually a power of two Linux allows for block accesses of any size, and user-space access is quite similar to character devices Underlying kernel data interface is very different from the character device interface

18. Network Interfaces Sends and receives data packets over communications media Handles interrupts pertaining to network traffic Contains kernel buffers to store packets that are waiting to be sent or have been received and not read In FreeBSD, network interfaces are considered to be block drivers

19. Linking Into Kernel Device drivers are created as discrete units of code known as “modules” Modules may be statically or dynamically linked into kernel Static linkage occurs at kernel compile time Dynamic linkage occurs at any time when the kernel is running We'll focus on dynamically linked modules

20. Dynamically Linked Driver Modules Modules are loaded using the insmod and rmmod commands If a module relies on another module being loaded first, use modprobe instead of insmod to automatically load the other required modules Use the lsmod command to list all currently loaded dynamic modules

21. Basic Driver Kernel Functions Driver setup/cleanup: module_init() and module_exit() Parameters passed to driver: module_param() Initializing a module using module_init() will: Expose the driver as a file in /dev filesystem Expose the driver in the devfs filesystem The ioctl() function is used to control the driver from the file exposed in the filesystem

22. Android → ashmem (Shared Mem) A. Shetty, K. Patil Lowmem, OOM Killer Zeyu Liu, Shudan Shi Wake Locks A. Iyer, A. Ananthanarayanan Alarm Timers Xiao Zhang, C. Wang

23. ASHMEM (By AbhishekShetty And KaustubhPatil)

24. Description • ASHMEM stands for Anonymous SHaredMEMory • Used to share memory between different process and manages the sharing in kernel levels. • The memory is virtual, not physical contiguous • It is an android shared memory addon to Linux kernel

25. Description Contd…. • It is a better support for low memory devices, because it can discard shared memory units under memory pressure. • Ashmemhas a reference counting, which keeps track of how many processes use the same area. • This area will not be removed until all the processes that hold this area have released it.

26. Pinning and Unpinning • ashmem introduces the concept of page pinning. • Pinned pages (the default) behave like any anonymous memory. • Unpinned pages are available to the kernel for eviction. • When re-pinning the pages, the return value instructs user-space as to any eviction. • In this manner, user-space processes may implement caching and similar resource management that efficiently integrates with kernel memory management.

27. IOCTL To use this, programs open /dev/ashmem, use mmap() on it, and can perform one or more of the following ioctls: • ASHMEM_SET_NAME • ASHMEM_GET_NAME • ASHMEM_SET_SIZE • ASHMEM_GET_SIZE • ASHMEM_SET_PROT_MASK • ASHMEM_GET_PROT_MASK • ASHMEM_PIN • ASHMEM_UNPIN • ASHMEM_GET_PIN_STATUS • ASHMEM_PURGE_ALL_CACHES

28. Libcutils InterfaceFunction calls • Instead of opening the region using the same name, for security reasons the file descriptor is passed to the other process via binder IPC. • The libcutils interface for ashmem consists of the following calls: (found in system/core/include/cutils/ashmem.h) • intashmem_create_region(const char *name, size_t size); • intashmem_set_prot_region(intfd, intprot); • intashmem_pin_region(intfd, size_t offset, size_tlen); • intashmem_unpin_region(intfd, size_t offset, size_tlen); • intashmem_get_size_region(intfd);

29. ASHMEM v/s PMEM • ashmemand pmem are very similar. Both are used for sharing memory between processes. • ashmem uses virtual memory, whereas pmem uses physically contiguous memory. One big difference is that with ashmem, you have a ref-counted object that can be shared equally between processes. • Example, two processses two processes sharing ashmem memory buffer. This buffer goes away when both processes have removed all their refrences by closing all their file descriptors. • However , in case of pmem it is different, as it needs to maintains a physical memory to virtual memory mapping. As a result, the process that allocates pmem heap is required to hold the file descriptor until all the other refrences are closed.

30. Choice between ASHMEM & PMEM • The choice between ashmem and pmem depends on whether you  need physically contiguous buffers. • Example, in the device G1 [T-Mobile's Android phone], the hardware 2D engineis used to do scaling, rotation, and color conversion, so pmem heaps were required. • However its emulator doesn't have a pmem driver and doesn't really need one, so ashmem is used in th eemulator. • If ashmem is used on the G1, it results in the loss of the hardware 2D engine capability.

31. ALL THE BEST For Your FINALS

32. Android ashmem (Shared Mem) A. Shetty, K. Patil → Lowmem, OOM Killer Zeyu Liu, Shudan Shi Wake Locks A. Iyer, A. Ananthanarayanan Alarm Timers Xiao Zhang, C. Wang

33. Lowmemory and OOM Killer CIS 657

34. What is OOM • OOM stands for Out Of Memory. • Happened when VM(MM) has no enough pages. • It may caused by page fault or allocated some pages. • The last situation when kernel want to see happens.

35. How Linux Handle OOM • Linux has an entry (function) called out_of_memory handle for this. • Located in linux_kernel/mm/oom_kill.c • Traditional out_of_memory killer is identical between Linux and Android • But android has a specify handler called Low Memory Killer

36. What is OOM do • Kill the biggest process (normally) • Has some heuristics to decide which process to kill • Usually the process has the biggest memory will be the victim (been killed) • Not the system init process, not prefer root’s process (guaranteed!)

37. Organization of the OOM killer • The function which does the actual scoring of a process in the effort to find the best candidate for elimination is called badness(), which results from the following call chain: • _alloc_pages -> out_of_memory() -> select_bad_process() -> badness()

38. Calculating Point • The scoring of a process starts with the size of its resident memory • The independent memory size of any child (except a kernel thread) will be added to the score • ‘Niced’(less important) processes have their scores increased, and long running processes have their scores decreased

39. Processes with CAP_SYS_ADMIN and CAP_SYS_RAWIO each flag have their scores reduced Finally the accumulated score is bitshifted by the user-settable value of /proc/<pid>/oomadj Non privileged process which together with its children uses lots of memory, has been niced, and does no raw I/O will be killed Calculating Point

40. Android Low Memory Killer • Android’s specific memory protection method. • Design for limited resource device. • Separate process class with System Level, User Visual Level, and Long Time No Use Level. • Only execute when memory is low

41. Android running processes categories • FOREGROUND_APP • VISIBLE_APP • SECONDARY_SERVER • HIDDEN_APP • CONTENT_PROVIDER • EMPTY_APP

42. #cat/sys/module/lowmemorykiller/parame ers/minfree Output :1536,2048,4096,5120,5632,6144 These values are the 6 memory limits on which Anedroid starts to kill processes of one of the 6 categories above when the memory limits below the threshold

43. Android ashmem (Shared Mem) A. Shetty, K. Patil Lowmem, OOM Killer Zeyu Liu, Shudan Shi → Wake Locks A. Iyer, A. Ananthanarayanan Alarm Timers Xiao Zhang, C. Wang

44. Wake Locks -Apoorva Iyer--Ananthanarayanan, Ajai Sankaranarayanan-

45. Wake Locks • Power Management System Class • Manages suspend and wake up • interface -> /sys/power/wake_lock

46. Wake Locks (contd..) • Prevent Suspend. • Power consuming feature. • android.permission.WAKE_LOCK - Cautious Usage!

47. Wake Locks - In Linux • Initialize wake lock to be used in the program. • Types, • WAKE_LOCK_SUSPEND • WAKE_LOCK_IDLE • Proc file for the list of wakelocks, /proc/wakelocks.

48. Wake Locks - In Linux (contd..) • Wake_lock(). • Wake_unlock()

49. Wake Locks - In Linux(contd..) • wake_lock_timeout(lock,HZ); • Release the wake lock after an interval. • wake_lock_destroy(lock);  • Called before freeing the memory.

50. Two Lists • Inactive • List of released wakelocks • Active • List of acquired wakelocks.