Integrating Portable and Distributed Storage

1. Integrating Portable and Distributed Storage Niraj Tolia, Jan Harkes, Michael Kozuch, and M. Satyanarayanan CMU and Intel Research Pittsburgh USENIX FAST 2004 Presenter: Yongjoon Son 2005. 11. 23

2. One-line Comment • This paper describes a technique called lookaside caching to integrate portable storage devices into the distributed file systems.

3. Contents • Motivation • Use Scenario • Lookaside Caching • Lookaside Caching Flowchart • Implementation • Evalution • Kernel Compile • Internet Suspend/Resume • Trace Replay • Broader Use of Lookaside Caching • Discussion

4. Motivation • Sneakernet • “Transfer of electronic information by physically carrying removable media from one computer to another” – Wikipedia • Why is it alive and well today in spite of advances in networking and distributed file systems? • Full confidence that you will be ableto access that data anywhere,regardless of network quality,network or server outages,and machine configuration. • The capacity and the performance ofthe portable storage are ever-improving. • Limitations • Ensuring right device? current version? • Guarding against loss, theft and damage • Some chores

5. Integration Value

6. Use Scenario • You don’t ensure where you can connect network in your destination. You decide to bring portable devices. • If the network doesn’t work in the destination, you use the device directly. Though the data can be stale, you can access it. • Otherwise, you can access the up-to-date data through the dfs. But, the portable devices can still work as a cache. • On a slow network or with a heavily loaded server, you will benefit from improved performance. • With network or server outages, you will benefit from improved availability if your dfs supports disconnected operation and if you have hoarded all your meta-data.

7. Lookaside Caching • Once a client possesses valid meta-data for an object, it looks up the corresponding file content in the mounted portable storage devices before accessing the file server. • Minimal disruption of existing usage model • Based upon AFS2-style whole-file caching • Lookaside caching extends the definition of meta-data to include a cryptographic hash of data contents. • Additional 20 bytes if SHA-1 is used as the hash Client Program Venus (cache mgr) Vice (servers) network User level Return from syscall system call Lookaside indexes Coda FS Ext2 FS ISO 9660 FS NFS VFS Portable Storage Kernel

8. Lookaside Caching Flowchart open() fetch attributes Yes cached No network fetch done done

9. Lookaside Caching Flowchart open() fetch attributes + hash Yes cached No network fetch done done

10. Lookaside Caching Flowchart open() fetch attributes + hash Yes cached No portable done

11. Lookaside Caching Flowchart open() fetch attributes + hash Yes cached No Yes portable done done

12. Lookaside Caching Flowchart open() fetch attributes + hash Yes cached No Yes portable No done network fetch done done

13. Implementation • Implemented lookaside caching in the Coda file system on Linux. • The implementation consists of four parts • A small change to the client-server protocol • ViceGetAttr() -> ViceGetAttrPlusSHA() • ViceValidateAttrs() -> ViceValidateAttrsPlusSHA() • If the server doesn’t support lookaside caching, it falls back to using the original RPCs • A quick index check in code path for handling a cache miss • A tool for generating lookaside indexes • mkdb utility to generate index file • Index generated from normal file system tree • Lazy update process • Allows users to change data • A set of user commands to include or exclude specific lookaside devices

14. Lookaside Commands on Client • Dynamic inclusion or exclusion of lookaside devices is done through user-level commands. • Multiple lookaside devices can be in use at the same time. • The devices are searched in order of inclusion

15. Evaluation • The performance of the lookaside caching depends on • The workload • The network quality • The overlap between data on the lookaside device and data accessed from the distributed file system • Three different benchmarks • Kernel compile benchmark • Virtual machine migration benchmark (ISR) • Single-user trace replay benchmark

16. Kernel Compile • Benchmark Description • The kernel being compiled is version 2.4.18. • The kernel on the lookaside device varied Key characteristics of the Linux kernel versions used in the compilation benchmark

17. Kernel Compile • Experiment Setup • Client • 3.0 GHz Pentium 4 processor, 2GB SDRAM • File cache size was large enough to prevent eviction during the experiments • Operated in write-disconnected mode • The client file cache was always cold at the start of an experiment. To discount the effect of a cold I/O buffer cache on the server, a warming run was done prior to each set of experiments. • File Server • 2.0 GHz Pentium 4 processor, 1GB SDRAM • Both ran Red Hat 9.0 Linux and Coda 6.0.2, and were connected by 100 Mb/s Ethernet

18. Kernel Compile • Experiment Setup • Run at four different bandwidth settings • 100 Mb/s, 10 Mb/s, 1 Mb/s and 100 Kb/s • NISTNet network router to control bandwidth • No extra latency was added at 100 Mb/s and 10 Mb/s. For 1 Mb/s and 100 Kb/s, add 10 ms and 100 ms respectively • Lookaside device • 512 MB Hi-Speed USB flash memory keychain. • Manufacturer claim: read 48 Mb/s, write 36 Mb/s Measured portable storage device performance

19. Kernel Compile • Result • The performance metric is the elapsed time to compile the 2.4.18 kernel. • Negative performance improvement means that the overhead of lookaside caching exceeds its benefit. • Since the client cache manager already monitors bandwidth to servers, it would be simple to suppress lookaside at high bandwidths.

20. Internet Suspend/Resume • Benchmark description • The ISR prototype layers VMware on Coda, and represents VM state as a tree of 256 KB files. • Part of the VM state can be copied to the device at suspend. • Common Desktop Application (CDA) • Modeling an interactive Windows user. • Visual Basic scripting to drive Microsoft Office applications such as Word, Excel, Powerpoint, Access and Internet Explorer. • 113 independently-timed operations such as find-and-replace, open-document, and save-as-html.

21. Internet Suspend/Resume • Experiment Setup • Clients • 2.0 GHz Pentium 4 processor, 1GB RAM, Red Hat Linux 7.3 Linux • VMware Workstation 3.1, 8GB Coda file cache • VM is configured to have 256 MB of RAM and 4GB of disk, and runs Windows XP as the guest OS • Server • 1.2 GHz Pentium III Xeon processor, 1G RAM, Red Hat Linux 7.3 Linux • 100 Mb/s Ethernet, NISTNet network emulator • A USB flash memory keychain is updated at suspend with the minimal state needed for resume. This is a single 41 MB file corresponding to the compressed physical memory image of the suspended virtual machine.

22. Internet Suspend/Resume • Results • Performance metrics • Resume Latency (How slow is the resume step?) • Total Operation Latency (After resource, how much is work slowed?) End of Benchmark VM started VM usable Resume Latency Total Operation Latency Time Resume Latency1 Total Operation Latency2 1 A single 41 MB file corresponding to the compressed physical memory image of the suspended virtual machine (USB flash memory) 2 The VM state captured after installation of Windows XP and the Microsoft Office suite (DVD)

23. Trace Replay • Benchmark Description Update Ops: the percentage of operations that change the file system sate such as mkdir, close-after-write, etc. Working Set: the size of the data accessed during trace execution • Experiment Setup • Same as the Kernel Compile Experiment

24. Trace Replay • Results • Performance metric • The time taken for trace replay completion • Vary the amount of data found on the device The overhead of lookaside caching dominates Due to the large number of meta-data accesses

25. Broader Use of Lookaside Caching • Content-Addressable Storage (CAS) • Experiment • Extend the prototype implementation • For the ISR benchmark, evaluate the performance benefit of using a LAN-attached CAS provider with same contents as the DVD • CAS provider is on a faster machine than the file server • Cooperative Caching • A collection of dfs clients with mutual trust can export each other’s file caches as CAS providers.

26. Discussion • Internet Suspend/Resume VM state VM state Part of VM state Part of VM state Lookaside Caching SoulPad Internet Suspend/Resume If you merchandise, which model do you prefer?