html5-img
1 / 64

Disk Storage, Basic File Structures, and Hashing

Disk Storage, Basic File Structures, and Hashing. Introduction. In a computerized database, the data is stored on computer storage medium, which includes: Primary Storage can be processed directly by the CPU e.g., the main memory, cache fast, expensive, but of limited capacity

malana
Download Presentation

Disk Storage, Basic File Structures, and Hashing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Disk Storage, Basic File Structures, and Hashing

  2. Introduction • In a computerized database, the data is stored on computer storage medium, which includes: • Primary Storage • can be processed directly by the CPU • e.g., the main memory, cache • fast, expensive, but of limited capacity • Secondary Storage • cannot be processed directly by the CPU • magnetic disks, optical disks, tapes • slow, cost less, but have a large capacity.

  3. Storage Hierarchy $$ Volatile Cache Primary storage unit price Memory Flash Memory Secondary storage Magnetic Disk speed Non-volatile Tertiary storage Optical Disk Magnetic Tape

  4. Storage of Databases • For the following reasons, most databases are stored permanently on secondary storage: • They are too large to fit entirely in main memory • They must persist over long period of times, but the main memory is a volatile storage • Secondary storage costs less

  5. Secondary Storage • Magnetic-disk: cannot be directly processed by the CPU; it must be brought to the main memory first. • Data is stored on spinning disk, and read/written magnetically • Primary medium for the long-term storage of data; typically stores entire database. • Non-volatile • slow access to data • large storage capacity (on the order of gigabytes)

  6. Disk Storage Devices • Preferred secondary storage device for high storage capacity and low cost. • Data stored as magnetized areas on magnetic disk surfaces. • A diskpack contains several magnetic disks connected to a rotating spindle. • Disks are divided into concentric circular tracks on each disk surface. • Track capacities vary typically from 4 to 50 Kbytes or more

  7. Disk Storage Devices (contd.) • A track is divided into smaller blocks or sectors • because it usually contains a large amount of information • The division of a track into sectors is hard-coded on the disk surface and cannot be changed. • One type of sector organization calls a portion of a track that subtends (faces) a fixed angle at the center as a sector. • A track is divided into blocks. • The block size B is fixed for each system. • Typical block sizes range from B=512 bytes to B=4096 bytes. • Whole blocks are transferred between disk and main memory for processing.

  8. Disk Storage Devices (contd.) • A read-write head moves to the track that contains the block to be transferred. • Disk rotation moves the block under the read-write head for reading or writing. • A physical disk block (hardware) address consists of: • a cylinder number (imaginary collection of tracks of same radius from all recorded surfaces) • the track number or surface number (within the cylinder) • and block number (within track). • Reading or writing a disk block is time consuming because of the seek time s and rotational delay (latency) rd. • Double buffering can be used to speed up the transfer of contiguous disk blocks.

  9. Physical Characteristics of Disks

  10. Components of a Disk • The platters spin (say, 90rps). • The arm assembly is moved in or out to position a head on a desired track. • Read-writehead • Positioned very close to the platter surface (almost touching it) • Reads or writes magnetically encoded information. • Only one head reads/writes at any one time. • Surface of platter divided into circular tracks

  11. Physical Characteristics of Disks • Track • an information storage circle on the surface of a disk. • Over 16,000 tracks per platter • each track can store between 4KB and 50KB of data. • Each track is divided into sectors. • Tracks under heads make a cylinder (imaginary!) • Cylinder • the tracks with the same diameter on all surfaces of a disk pack. • Cylinder i consists of i-th track of all the platters

  12. Physical Characteristics of Disks • Sector • a part of a track with fixed size • separated by fixed-size interblock gaps • Typical sectors per track • 200 (on inner tracks) to 400 (on outer tracks)

  13. Sectors

  14. Disk I/O Model of Computation • Disk I/O is equivalent to one read or write operation of a single block • It is very expensive compared with what is likely to be done once the block gets in main memory • one random disk I/O ~ about 1,000,000 machine instructions in terms of time • Cost for computation that requires secondary storage is computed only by disk I/Os.

  15. Pages and Blocks • Data files decomposed into pages (blocks) • fixed size piece of contiguous information in the file • sizes range from 512 bytes to several kilobytes • block is the smallest unit for transferring data between the main memory and the disk. • Address of a page (block): • (cylinder#, track# (within cylinder), sector# (within track)

  16. Pages and Blocks Track Sector Gap ... 1 3 4 2 One track 1 page/block = 4 Sectors

  17. Page I/O • Page I/O --- one page I/O is the cost (or time needed) to transfer one page of data between the memory and the disk. • The cost of a (random) page I/O = • seek time + rotational delay + block transfer time • Seek time • time needed to position read/write head on correct track. • Rotational delay (latency) • time needed to rotate the beginning of page under read/write head. • Block transfer time • time needed to transfer data in the page/block.

  18. Page I/O • Average rotational delay (rd) • rd = ½ * (1/p) min = (60*1000)/(2*p) msec • OR • rd = ½ * cost of 1 revolution • = ½ * (60*1000/p) msec • where • p is speed of disk rotation (how many revolutions per minute - rpm) • Example • Speed of disk rotatioon is p = 3600 rpm • 60 revolutions/sec • 1 rev. = 16.66 msec. (1 second = 1000 msec) • rd = 8.33 ms

  19. Page I/O • Transfer rate (tr) • tr = track size / cost of one revolution • = track size / (60*1000/p) in msec • Bulk transfer rate (btr) • btr = (B/(B+G)) * tr bytes/msec • Where B is the block size in bytes • G is interblock gap size in bytes • Block transfer time (btt) • btt = B / tr not taking into acount G • btt = B / btr taking into acount G

  20. Page I/O • Example: • Track size = 50 KB and p = 3600 rpm • Block size B = 3KB = 3000 bytes • tr = (50*1000)/(60*1000/3600) = 3000 bytes/msec • btt = B / tr = 3000/3000 = 1 msec

  21. Page I/O • Average time for reading/writing nconsecutive pages that are in the same track or cylinder = s + rd + n* btt • Average time for reading/writing consecutivelynnoncontigues pages/blocks that are in the same cylinder = s + n* (rd + btt)

  22. An Example • A hard disk specifications: • 4 platters, 8 Surfaces, 3.5 Inch diameter • 213 = 8192 tracks/surface • 28 = 256 sectors/track • 29 = 512 bytes/sector • Average seek time s = 25 ms • Rotation rate rd = 3600 rpm = 60 rps • 1 rev. = 16.66 msec • Transfer rate • tr = 1 KB in 0.117 ms • tr = 1 KB in 0.130 ms with gap

  23. An Example • What is the total capacity of this disk • 8 GB (8*213*28*29=233) • How many bytes does one track hold? • 256 sectors/track*512 bytes/sector = 128KB • How many blocks per track? • one block = 4096 bytes = 8 sectors (4096/512) • 256/8 = 32 blocks/track

  24. An Example • How long does it take to access one block? • One block = 4096 bytes • 8 sectors = 4096/512 • Rotation rate r • 1 rev. = 16.66 msec. • Time to access 1 sector (s + r/2 + tr/(secters/KB) • 25 + (16.66/2) + .117/2 = 33.3885 ms. • time to access 1 block • time to access the first sector of the block + time to access the subsequent 7 sectors.

  25. An Example 1 ... 8 3 2 1 block = 8 Sectors • T = 25 + (16.66/2) + (0.117/2) * 1 + (0.13/2) *7 = 33.3885 + 0.455 ms = 33.8435ms • Compare to one sector access time: 33.3885 ms

  26. Buffering • A buffer • is a contiguous reserved area in main memory available for storage of copies of disk blocks. • to speed up the processes. • For a read command • the block from disk is copied into the buffer. • For a write command • the contents of the buffer are copied into the disk.

  27. Accessing Data Through RAM Buffer RAM Block transfer Buffer Application block Record transfer Page frames

  28. Buffer Manager • Programs call on the buffer manager when they need a block from disk. • If the block is already in the buffer, • the requesting program is given the address of the block in main memory • If the block is not in the buffer, • the buffer manager allocates space in the buffer for the block, replacing (throwing out) some other block, if required, to make space for the new block. • The block that is thrown out is written back to disk only if it was modified since the most recent time that it was written to/fetched from the disk.

  29. Buffer Manager • Once space is allocated in the buffer, the buffer manager reads the block from the disk to the buffer, and passes the address of the block in main memory to requester. • Buffer Replacement Policy: • Frame is chosen for replacement by a replacement policy: • Least-recently-used (LRU), MRU, FIFO, etc. • Policy can have big impact on # of I/O’s; depends on the access pattern.

  30. File Organization • The database is stored as a collection of files. • Each file is a sequence of records. • A record is a sequence of fields. • Records are stored on disk blocks. • A file can have fixed-length records or variable-length records.

  31. File Organization • Fixed length records • Each record is of fixed length. Pad with spaces. • Variable length records • different records in the file have different sizes. • Arise in database systems in several ways: • different record types in a file. • same record type with (variable-length fields, repeating field, or optional fields)

  32. File Organization

  33. Fixed-Length Records • Insertion: • Store record i starting from byte n  (i – 1), where n is the size of each record. • Deletion of record i: • Packed format: • move records i + 1, . . ., n to i, . . . , n – 1 • OR • move record n to i • Unpacked format (do not move records, but) • link all free records on a free list • OR • Use bitmap vector

  34. Free Lists • Store the address of the first deleted record in the file header. • Use this first record to store the address of the second deleted record, and so on.

  35. Page Formats: Fixed Length Records Slot 1 Slot 1 Slot 2 Slot 2 Free Space . . . . . . Slot N Slot N Slot M N . . . 1 1 1 M 0 M ... 3 2 1 number of records number of slots PACKED UNPACKED, BITMAP • Record id = <page id, slot #>.

  36. Variable-Length Records Represenation • Byte-String representation • Attach an end-of-record () control character to the end of each record • Difficulty with deletion and growth • Slotted-page header contains: • number of record entries • location and size of each record • end of free space in the block

  37. Slotted Page Structure • Records can be moved around within a page to keep them contiguous with no empty space between them • entry in the header must be updated. • Pointers should not point directly to record - instead they should point to the entry for the record in header.

  38. Fixed-Length Representation • Reserved Space • can use fixed-length records of a known maximum length • unused space in shorter records filled with a null or end-of-record symbol.

  39. Fixed-Length Representation • List Representation by Pointers • A variable-length record is represented by a list of fixed-length records, chained together via pointers. • Can be used even if the maximum record length is not known

  40. Fixed-Length Representation • Disadvantage: space is wasted in all records except the first in a a chain. • Solution is to allow two kinds of block in file: • Anchor block: contains the first records of chain • Overflow block: contains records other than those that are the first records of chairs.

  41. Blocking Factor • Blocking Factor (bfr) - the number of records that can fit into a single block. • bfr = ⌊B/R⌋ • B : Block size in bytes • R: Record size in bytes • Example: • Record size R = 100 bytes • Block Size B = 2,000 bytes • Thus the blocking factor bfr = 2000/100 = 20 • The number of blocks b needed to store a file of rrecords: • b = r/bfr blocks

  42. Spanned & Unspanned Records • A block is the unit of data transfer between disk and memory. • Unspanned records: • A record is found in one and only one block. • records do not span across block boundaries. • Used with fixed-length records having BR • Spanned records: • Records are allowed to span across block boundaries. • Used with variable-length records having RB • In variable-length records, either organization can be used.

  43. Placing File Records on Disk • A file header or file descriptor contains information about a file (e.g., the disk address, record format descriptions, etc.)

  44. Allocating File Blocks on Disk • The physical disk blocks that are allocated to hold the records of a file can be contiguous, linked, or indexed. • In contiguous allocation, the file blocks are allocated to consecutive disk blocks. • In linked allocation, each file block contains a pointer to the next file block. • In indexed allocation, one or more index blocks contain pointers to the actual file blocks.

  45. Organization of Records in Files • Heap File Organization • a record can be placed anywhere in the file where there is space, or at the end • for full file scans or frequent updates • Data unordered (unsorted) • Sorted/OrderedFile Organization • store records sorted in order, based on the value of the search key of each record • Need external sort or an index to keep sorted • HashingFile Organization • a hash function computed on some attribute of each record • the result specifies in which block of the file the record should be placed

  46. Heap File Organization • Records are placed in the file in the order in which they are inserted. Such an organization is called a heap file. • Insertion is at the end • takes constant timeO(1) (very efficient) • Searching • requires a linear search (expensive) • Deleting • requires a search, then delete • Select, Update and Delete • take b/2 time (linear time) in average • b is the number of blocks

  47. Heap File Organization • For a file of unorderedfixed-length records using unspanned blocks and contiguous allocation, it is straightforward to access any record by its position in the file. • If the records are numbered 0,1,2, …, r-1 and • The records in each block are numbered 0,1,2, …, f-1, where f is the blocking factor • The the i-th record of the file is located in • Blocki/f and in the • (i mod f)-th record in that block

  48. Heap File Organization • A Heap file allows us to retrieve records: • by specifying the rid, or • by scanning all records sequentially • Accessing a record by its position does not help locate a record based on a search condition.

  49. File Stored as a Heap File 666666 MGT123 F1994 4.0 123456 CS305 S1996 4.0 page 0 987654 CS305 F1995 2.0 717171 CS315 S1997 4.0 666666 EE101 S1998 3.0 page 1 765432 MAT123 S1996 2.0 515151 EE101 F1995 3.0 234567 CS305 S1999 4.0 page 2 878787 MGT123 S1996 3.0

More Related