DeFiler interfaces

1. DeFiler interfaces create, destroy, read, write a dfile list dfiles, sync() DFS read(), write() startFetch(), startPush() waitValid(), waitClean() isBusy() DBuffer dbuf = getBlock(blockID) releaseBlock(dbuf) sync() DBufferCache DBuffer ioComplete() blockID = getBlockID() buf[] = getBuffer() startRequest(dbuf, r/w) VirtualDisk

2. DeFiler interfaces: overview create, destroy, read, write a dfile list dfiles DFS read(), write() startFetch(), startPush() waitValid(), waitClean() DBuffer dbuf = getBlock(blockID) releaseBlock(dbuf) DBufferCache DBuffer ioComplete() startRequest(dbuf, r/w) VirtualDisk

3. DFS /* creates a new dfile and returns the DFileID */ public DFileID createDFile(); /* destroys the dfile named by the DFileID */ public void destroyDFile(DFileID dFID); /* reads contents of the dfile named by DFileID into the ubuffer * starting from ubuffer offset startOffset; at most count bytes are transferred */ public int read(DFileID dFID, byte[] ubuffer, int startOffset, int count); /* writes to the file named by DFileID from the ubuffer * starting from ubuffer offset startOffset; at most count bytes are transferred */ public int write(DFileID dFID, byte[] ubuffer, int startOffset, int count); /* List DFileIDs for all existing dfiles in the volume */ public List<DFileID> listAllDFiles();

4. Small details: low-level I/O DBuffer required interfaces VirtualDisk needs basic info about a DBuffer for I/O: the dbuf’s blockID and a reference to its byte buffer. blockID = getBlockID() buf[] = getBuffer() VirtualDisk has private methods to “format” the “disk” (VDF file), and fetch/push blocks at specified blockIDs. We give you sample code for these functions. VirtualDisk

5. Small details: initializing new DFS(format) Initialization starts with a call from the test program. DFS If DFS constructor call has format == true, then all data in VDF is discarded and zeroed. constructor (int numblocks) DBuffer DBufferCache Create numblocks DBuffer objects (dbufs) and memory buffer regions (of size blocksize) for those dbufs. constructor (boolean format) VirtualDisk Create/truncate VDF (w/ optional name), or open existing VDF if format == false.

6. Small details: exiting sync DFS A test program should call DFS sync before exit(), to force any dirty blocks in the I/O cache out to disk. Sync is implemented in DBufferCache, and is synchronous: don’t return until all writes complete. sync DBufferCache VirtualDisk

7. Big details: DBuffer DFS read(…) write(...) startFetch(), startPush() waitValid(), waitClean() A DBuffer dbuf returned by getBlock is always associated with exactly one block in the disk volume. But it might or might not be “in sync” with the underlying disk contents. DBuffer A dbuf is valid iff it has the “correct” copy of the data. A dbuf is dirty iff it is valid and has an update (a write) that has not yet been written to disk. A valid dbuf is clean if it is not dirty. Your DeFiler should return only valid data to a client. That may require you to zero the dbuf or fetch data from the disk. Your DeFiler should ensure that all dirty data is eventually pushed to disk.

8. DBufferCache /* Get buffer for block specified by blockID The buffer is “held” until the caller releases it. A “held” buffer cannot be evicted: its block ID cannot change. */ public DBuffer getBlock(int blockID); /* Release the buffer so that it may be eligible for eviction. */ public void releaseBlock(DBuffer dbuf); /* Write back all dirty blocks to the volume, and wait for completion. */ public void sync();

9. DBuffer /* Start an asynchronous fetch of associated block from the volume */ public void startFetch(); /* Start an asynchronous write of buffer contents to block on volume */ public void startPush(); /* Check whether the buffer has valid data*/ public boolean checkValid(); /* Wait until the buffer has valid data (i.e., wait for fetch to complete) */ public boolean waitValid(); /* Check whether the buffer is dirty, i.e., has modified data to be written back */ public boolean checkClean(); /* Wait until the buffer is clean (i.e., wait for push to complete) */ public boolean waitClean(); /* Check if buffer is evictable: not evictable if I/O in progress, or buffer is held. */ public boolean isBusy();

10. DBuffer /* Reads into the ubuffer[ ] from the contents of this Dbuffer dbuf. * Check first that dbuf has a valid copy of the data! * startOffset is for the ubuffer, not for dbuf. * Reads begin at offset 0 in dbuf and move at most count bytes. */ public int read(byte[] ubuffer, int startOffset, int count); /* Writes into this Dbuffer dbuf from the contents of ubuffer[ ]. * Mark dbuf dirty! startOffset is for the ubuffer, not for dbuf. * Writes begin at offset 0 in dbuf and move at most count bytes. */ public int write(byte[] ubuffer, int startOffset, int count); These calls are for use by the DFS layer to read/write user data between client ubuffers and Dbufferdbufs. DFS may read/write only on a helddbuf.

11. VirtualDisk /* * Start an asynchronous I/O request to the device/disk. * The blockID and buffer array are given by the DBuffer dbuf. * The operation is either READ or WRITE (DiskOperationType). */ public void startRequest(DBuffer dbuf, DiskOperationType rw) throws…;

12. Big issues: caching DeFiler uses an I/O cache in memory to stage transfers to/from disk and to reduce the need for I/O. The cache has a set of DBufferdbuf buffer objects. Each dbuf is either free or it is associated with exactly one disk block blockID. I/O to/from a block is staged from its dbuf. Each block has at most one dbuf. The dbuf for a block is kept in cache after access. If a requested block is already resident in the cache, then getBlock finds its dbuf and returns it. Else it allocates a free dbuf for the block. The system discards (evicts) a cached block if it has a better use for the memory. It frees the evicted block’s dbuf and soon reuses the dbuf for some other block. DFS dbuf = getBlock(blockID) releaseBlock(dbuf) DBufferCache DBuffer

13. Big issues: eviction The I/O cache system has a replacement policy to select candidate blocks for eviction. It keeps an evict pool of dbufs ordered by some measure of their suitability for eviction, e.g., Least Recently Used (LRU). The evict pool data structure may require more state in dbufs, or interactions between DBufferCache and DBuffer. This is up to you: no formats or interfaces are specified. DFS dbuf = getBlock(blockID) releaseBlock(dbuf) DBufferCache DBuffer startRequest(dbuf, r/w); ioComplete() VirtualDisk

14. Buffer states DBufferCache must not evict a block when its dbuf is in use by the layer above or below. You must think carefully about DBuffer (dbuf) states and how to synchronize access to dbufs. This is up to you. Suggestion. A dbuf is pinned if I/O is in progress, i.e., a VDF request has started but not yet completed. A dbuf is held if DFS obtained a reference to the dbuf from getBlock but has not yet released the dbuf. Don’t evict a dbuf that is pinned or held: pick another candidate. DFS dbuf = getBlock(blockID) releaseBlock(dbuf) DBufferCache DBuffer startRequest(dbuf, r/w); ioComplete() VirtualDisk