Review of database systems concepts chapters 1 2 3 4 5 8 9 in r
This presentation is the property of its rightful owner.
Sponsored Links
1 / 124

Review of Database Systems Concepts Chapters 1,2,3,4,5,8,9 in [R] PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on
  • Presentation posted in: General

Review of Database Systems Concepts Chapters 1,2,3,4,5,8,9 in [R]. [R] = Ramakrishnan & Gehrke: Database Management Systems, Third Edition. What Is a DBMS?. A very large, integrated collection of data. Models real-world enterprise. Entities (e.g., students, courses)

Download Presentation

Review of Database Systems Concepts Chapters 1,2,3,4,5,8,9 in [R]

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Review of database systems concepts chapters 1 2 3 4 5 8 9 in r

Review of Database Systems ConceptsChapters 1,2,3,4,5,8,9 in [R]

[R] = Ramakrishnan & Gehrke: Database Management Systems, Third Edition


What is a dbms

What Is a DBMS?

  • A very large, integrated collection of data.

  • Models real-world enterprise.

    • Entities (e.g., students, courses)

    • Relationships (e.g., Madonna is taking CS564)

  • A Database Management System (DBMS)is a software package designed to store and manage databases.


Files vs dbms

Files vs. DBMS

  • Application must stage large datasets between main memory and secondary storage (e.g., buffering, page-oriented access, 32-bit addressing, etc.)

  • Special code for different queries

  • Must protect data from inconsistency due to multiple concurrent users

  • No Crash recovery

  • No Security and access control


Why use a dbms

Why Use a DBMS?

  • Data independence and efficient access.

  • Reduced application development time.

  • Data integrity and security.

  • Uniform data administration.

  • Concurrent access, recovery from crashes.


Why study databases

Why Study Databases??

?

  • Shift from computation to information

    • at the “low end”: scramble to webspace (a mess!)

    • at the “high end”: scientific applications

  • Datasets increasing in diversity and volume.

    • Digital libraries, interactive video, Human Genome project, EOS project

    • ... need for DBMS exploding

  • DBMS encompasses most of CS

    • OS, languages, theory, AI, multimedia, logic


Data models

Data Models

  • A data modelis a collection of concepts for describing data.

  • Aschemais a description of a particular collection of data, using the a given data model.

  • The relational model of datais the most widely used model today.

    • Main concept: relation, basically a table with rows and columns.

    • Every relation has a schema, which describes the columns, or fields.


Levels of abstraction

Levels of Abstraction

  • Many views, single conceptual (logical) schemaand physical schema.

    • Views describe how users see the data.

    • Conceptual schema defines logical structure

    • Physical schema describes the files and indexes used.

View 1

View 2

View 3

Conceptual Schema

Physical Schema

  • Schemas are defined using DDL; data is modified/queried using DML.


Example university database

Example: University Database

  • Logical schema:

    • Students(sid: string, name: string, login: string,

      age: integer, gpa:real)

    • Courses(cid: string, cname:string, credits:integer)

    • Enrolled(sid:string, cid:string, grade:string)

  • Physical schema:

    • Relations stored as unordered files.

    • Index on first column of Students.

  • External Schema (View):

    • Course_info(cid:string,enrollment:integer)


Data independence

Data Independence *

  • Applications insulated from how data is structured and stored.

  • Logical data independence: Protection from changes in logical structure of data.

  • Physical data independence: Protection from changes in physical structure of data.

  • One of the most important benefits of using a DBMS!


Why study the relational model

Why Study the Relational Model?

  • Most widely used model.

    • Vendors: IBM, Informix, Microsoft, Oracle, Sybase, etc.

  • “Legacy systems” in older models

    • E.G., IBM’s IMS

  • Recent competitor: object-oriented model

    • ObjectStore, Versant, Ontos

    • A synthesis emerging: object-relational model

      • Informix Universal Server, UniSQL, O2, Oracle, DB2


Relational database definitions

Relational Database: Definitions

  • Relational database:a set of relations

  • Relation: made up of 2 parts:

    • Instance : a table, with rows and columns. #Rows = cardinality, #fields = degree / arity.

    • Schema :specifiesname of relation, plus name and type of each column.

      • E.G. Students(sid: string, name: string, login: string, age: integer, gpa: real).

  • Can think of a relation as a setof rows or tuples (i.e., all rows are distinct).


Primary key constraints

Primary Key Constraints

  • A set of fields is a keyfor a relation if :

    1. No two distinct tuples can have same values in all key fields, and

    2. This is not true for any subset of the key.

    • Part 2 false? A superkey.

    • If there’s >1 key for a relation, one of the keys is chosen (by DBA) to be the primary key.

  • E.g., sid is a key for Students. (What about name?) The set {sid, gpa} is a superkey.

  • Every relation must have a key


Example instance of students relation

Example Instance of Students Relation

  • Cardinality = 3, degree = 5, all rows distinct

  • Do all columns in a relation instance have to

  • be distinct?


Relational query languages

Relational Query Languages

  • A major strength of the relational model: supports simple, powerful querying of data.

  • Queries can be written intuitively, and the DBMS is responsible for efficient evaluation.

    • The key: precise semantics for relational queries.

    • Allows the optimizer to extensively re-order operations, and still ensure that the answer does not change.


The sql query language

The SQL Query Language

  • Developed by IBM (system R) in the 1970s

  • Need for a standard since it is used by many vendors

  • Standards:

    • SQL-86

    • SQL-89 (minor revision)

    • SQL-92 (major revision)

    • SQL-99 (major extensions, current standard)


The sql query language1

The SQL Query Language

  • To find all 18 year old students, we can write:

SELECT *

FROM Students S

WHERE S.age=18

  • To find just names and logins, replace the first line:

SELECT S.name, S.login


Querying multiple relations

Querying Multiple Relations

SELECT S.name, E.cid

FROM Students S, Enrolled E

WHERE S.sid=E.sid AND E.grade=“A”

  • What does the following query compute?

Given the following instances of Enrolled and Students:

we get:


Creating relations in sql

Creating Relations in SQL

  • Creates the Students relation. Observe that the type (domain) of each field is specified, and enforced by the DBMS whenever tuples are added or modified.

  • As another example, the Enrolled table holds information about courses that students take.

CREATE TABLE Students

(sid: CHAR(20),

name: CHAR(20),

login: CHAR(10),

age: INTEGER,

gpa: REAL)

CREATE TABLE Enrolled

(sid: CHAR(20),

cid: CHAR(20),

grade: CHAR(2))


Adding and deleting tuples

Adding and Deleting Tuples

  • Can insert a single tuple using:

INSERT INTO Students (sid, name, login, age, gpa)

VALUES (53688, ‘Smith’, [email protected], 18, 3.2)

  • Can delete all tuples satisfying some condition (e.g., name = Smith):

DELETE

FROM Students S

WHERE S.name = ‘Smith’

  • Powerful variants of these commands are available; more later!


Integrity constraints ics

Integrity Constraints (ICs)

  • IC: condition that must be true for any instance of the database; e.g., domain constraints.

    • ICs are specified when schema is defined.

    • ICs are checked when relations are modified.

  • A legalinstance of a relation is one that satisfies all specified ICs.

    • DBMS should not allow illegal instances.

  • If the DBMS checks ICs, stored data is more faithful to real-world meaning.

    • Avoids data entry errors, too!


Primary and candidate keys in sql

Primary and Candidate Keys in SQL

  • Possibly many candidate keys(specified using UNIQUE), one of which is chosen as the primary key.

CREATE TABLE Enrolled

(sid CHAR(20)

cid CHAR(20),

grade CHAR(2),

PRIMARY KEY (sid,cid) )

  • “For a given student and course, there is a single grade.” vs. “Students can take only one course, and receive a single grade for that course; further, no two students in a course receive the same grade.”

  • Used carelessly, an IC can prevent the storage of database instances that arise in practice!

CREATE TABLE Enrolled

(sid CHAR(20)

cid CHAR(20),

grade CHAR(2),

PRIMARY KEY (sid),

UNIQUE (cid, grade) )


Foreign keys referential integrity

Foreign Keys, Referential Integrity

  • Foreign key : Set of fields in one relation that is used to `refer’ to a tuple in another relation. (Must correspond to primary key of the second relation.) Like a `logical pointer’.

  • E.g. sid is a foreign key referring to Students:

    • Enrolled(sid: string, cid: string, grade: string)

    • If all foreign key constraints are enforced, referential integrity is achieved, i.e., no dangling references.

    • Can you name a data model w/o referential integrity?

      • Links in HTML! (XML)


Foreign keys in sql

Foreign Keys in SQL

  • Only students listed in the Students relation should be allowed to enroll for courses.

CREATE TABLE Enrolled

(sid CHAR(20), cid CHAR(20), grade CHAR(2),

PRIMARY KEY (sid,cid),

FOREIGN KEY (sid) REFERENCESStudents )

Enrolled

Students


Enforcing referential integrity

Enforcing Referential Integrity

  • Consider Students and Enrolled; sid in Enrolled is a foreign key that references Students.

  • What should be done if an Enrolled tuple with a non-existent student id is inserted? (Reject it!)

  • What should be done if a Students tuple is deleted?

    • Also delete all Enrolled tuples that refer to it.

    • Disallow deletion of a Students tuple that is referred to.

    • Set sid in Enrolled tuples that refer to it to a default sid.

    • (In SQL, also: Set sid in Enrolled tuples that refer to it to a special value null, denoting `unknown’ or `inapplicable’.)

  • Similar if primary key of Students tuple is updated.


Referential integrity in sql

Referential Integrity in SQL

  • SQL/92 and SQL:1999 support all 4 options on deletes and updates.

    • Default is NO ACTION (delete/update is rejected)

    • CASCADE (also delete all tuples that refer to deleted tuple)

    • SET NULL / SET DEFAULT (sets foreign key value of referencing tuple)

CREATE TABLE Enrolled

(sid CHAR(20),

cid CHAR(20),

grade CHAR(2),

PRIMARY KEY (sid,cid),

FOREIGN KEY (sid)

REFERENCESStudents

ON DELETE CASCADE

ON UPDATE SET DEFAULT )


Where do ics come from

Where do ICs Come From?

  • ICs are based upon the semantics of the real-world enterprise that is being described in the database relations.

  • We can check a database instance to see if an IC is violated, but we can NEVER infer that an IC is true by looking at an instance.

    • An IC is a statement about all possible instances!

    • From example, we know name is not a key, but the assertion that sid is a key is given to us.

  • Key and foreign key ICs are the most common; more general ICs supported too.


The entity relationship model

The Entity-Relationship Model

Chapter 2


Overview of database design

Overview of Database Design

  • Conceptual design: (ER Model is used at this stage.)

    • What are the entities and relationships in the enterprise?

    • What information about these entities and relationships should we store in the database?

    • What are the integrity constraints or business rules that hold?

    • A database `schema’ in the ER Model can be represented pictorially (ER diagrams).

    • Can map an ER diagram into a relational schema.


Er model basics

name

ssn

lot

Employees

ER Model Basics

  • Entity: Real-world object distinguishable from other objects. An entity is described (in DB) using a set of attributes.

  • Entity Set: A collection of similar entities. E.g., all employees.

    • All entities in an entity set have the same set of attributes. (Until we consider ISA hierarchies, anyway!)

    • Each entity set has a key.

    • Each attribute has a domain.


Er model basics contd

name

ssn

lot

Employees

super-visor

Reports_To

ER Model Basics (Contd.)

  • Relationship: Association among two or more entities. E.g., Attishoo works in Pharmacy department.

  • Relationship Set: Collection of similar relationships.

    • An n-ary relationship set R relates n entity sets E1 ... En; each relationship in R involves entities e1 E1, ..., en En

      • Same entity set could participate in different relationship sets, or in different “roles” in same set.

since

name

dname

subor-dinate

ssn

budget

lot

did

Works_In

Employees

Departments


Review key constraints

since

name

dname

ssn

lot

Employees

Manages

Review: Key Constraints

  • Each dept has at most one manager, according to the key constrainton Manages.

budget

did

Departments

Translation to

relational model?

1-to-1

1-to Many

Many-to-1

Many-to-Many


Logical db design er to relational

name

ssn

lot

Employees

Logical DB Design: ER to Relational

  • Entity sets to tables:

CREATE TABLE Employees

(ssn CHAR(11),

name CHAR(20),

lot INTEGER,

PRIMARY KEY (ssn))


Relationship sets to tables

Relationship Sets to Tables

  • In translating a relationship set to a relation, attributes of the relation must include:

    • Keys for each participating entity set (as foreign keys).

      • This set of attributes forms a superkey for the relation.

    • All descriptive attributes.

CREATE TABLE Works_In(

ssn CHAR(11),

did INTEGER,

since DATE,

PRIMARY KEY (ssn, did),

FOREIGN KEY (ssn)

REFERENCES Employees,

FOREIGN KEY (did)

REFERENCES Departments)


Logical db design er to relational cont

Logical DB Design: ER to Relational (Cont.)

  • Not so simple!

  • Some attributes are mapped to a separate relation

  • Some relationships (1:1, !:many) are mapped to an extra attribute in the ‘1’ relation

  • See any standard DB book for details


Summary of conceptual design

Summary of Conceptual Design

  • Conceptual design follows requirements analysis,

    • Yields a high-level description of data to be stored

  • ER model popular for conceptual design

    • Constructs are expressive, close to the way people think about their applications.

  • Basic constructs: entities, relationships, and attributes (of entities and relationships).

  • Some additional constructs: weak entities, ISA hierarchies, and aggregation.

  • Note: There are many variations on ER model.


Formal relational query languages

Formal Relational Query Languages

  • Two mathematical Query Languages form the basis for “real” languages (e.g. SQL), and for implementation:

    • Relational Algebra: More operational, very useful for representing execution plans.

    • Relational Calculus: Lets users describe what they want, rather than how to compute it. (Non-operational, declarative.)


Example instances

Example Instances

R1

  • “Sailors” and “Reserves” relations for our examples.

  • We’ll use positional or named field notation, assume that names of fields in query results are `inherited’ from names of fields in query input relations.

S1

S2


Relational algebra

Relational Algebra

  • Basic operations:

    • Selection ( ) Selects a subset of rows from relation.

    • Projection ( ) Deletes unwanted columns from relation.

    • Cross-product( ) Allows us to combine two relations.

    • Set-difference ( ) Tuples in reln. 1, but not in reln. 2.

    • Union( ) Tuples in reln. 1 and in reln. 2.

  • Additional operations:

    • Intersection, join, division, renaming: Not essential, but (very!) useful.

  • Since each operation returns a relation, operationscan be composed! (Algebra is “closed”.)


Projection

Projection

  • Deletes attributes that are not in projection list.

  • Schema of result contains exactly the fields in the projection list, with the same names that they had in the (only) input relation.

  • Projection operator has to eliminate duplicates! (Why??)

    • Note: real systems typically don’t do duplicate elimination unless the user explicitly asks for it. (Why not?)


Selection

Selection

  • Selects rows that satisfy selection condition.

  • No duplicates in result! (Why?)

  • Schema of result identical to schema of (only) input relation.

  • Result relation can be the input for another relational algebra operation! (Operatorcomposition.)


Union intersection set difference

Union, Intersection, Set-Difference

  • All of these operations take two input relations, which must be union-compatible:

    • Same number of fields.

    • `Corresponding’ fields have the same type.

  • What is the schema of result?


Cross product

Cross-Product

  • Each row of S1 is paired with each row of R1.

  • Result schema has one field per field of S1 and R1, with field names `inherited’ if possible.

    • Conflict: Both S1 and R1 have a field called sid.

  • Renaming operator:


Joins

Joins

  • Condition Join:

  • Result schema same as that of cross-product.

  • Fewer tuples than cross-product, might be able to compute more efficiently

  • Sometimes called a theta-join.


Joins1

Joins

  • Equi-Join: A special case of condition join where the condition c contains only equalities.

  • Result schema similar to cross-product, but only one copy of fields for which equality is specified.

  • Natural Join: Equijoin on all common fields.


Division

Division

  • Not supported as a primitive operator, but useful for expressing queries like: Find sailors who have reserved allboats.

  • Let A have 2 fields, x and y; B have only field y:

    • A/B =

    • i.e., A/B contains all x tuples (sailors) such that for everyy tuple (boat) in B, there is an xy tuple in A.

    • Or: If the set of y values (boats) associated with an x value (sailor) in A contains all y values in B, the x value is in A/B.

  • In general, x and y can be any lists of fields; y is the list of fields in B, and x y is the list of fields of A.


Examples of division a b

Examples of Division A/B

B1

B2

B3

A/B1

A/B2

A/B3

A


Find names of sailors who ve reserved boat 103

  • Solution 2:

  • Solution 3:

Find names of sailors who’ve reserved boat #103

  • Solution 1:


Find names of sailors who ve reserved a red boat

  • A more efficient solution:

Find names of sailors who’ve reserved a red boat

  • Information about boat color only available in Boats; so need an extra join:

A query optimizer can find this, given the first solution!


Find sailors who ve reserved a red or a green boat

  • What happens if is replaced by in this query?

Find sailors who’ve reserved a red or a green boat

  • Can identify all red or green boats, then find sailors who’ve reserved one of these boats:

  • Can also define Tempboats using union! (How?)


Find the names of sailors who ve reserved all boats

Find the names of sailors who’ve reserved all boats

  • Uses division; schemas of the input relations to / must be carefully chosen:

  • To find sailors who’ve reserved all ‘Interlake’ boats:

.....


Other relational languages

Other Relational Languages

  • Relational calculus -tuple or domain (QBE)

  • SQL -you already know!


Summary

Summary

  • The relational model has rigorously defined query languages that are simple and powerful.

  • Relational algebra is more operational; useful as internal representation for query evaluation plans.

  • Several ways of expressing a given query; a query optimizer should choose the most efficient version.


Data storage review

Data Storage Review

Chapter 8


Disks and files

Disks and Files

  • DBMS stores information on (“hard”) disks.

  • This has major implications for DBMS design!

    • READ: transfer data from disk to main memory (RAM).

    • WRITE: transfer data from RAM to disk.

    • Both are high-cost operations, relative to in-memory operations, so must be planned carefully!


Why not store everything in main memory

Why Not Store Everything in Main Memory?

  • Costs too much. $1000 will buy you either 128MB of RAM or 7.5GB of disk today.

  • Main memory is volatile. We want data to be saved between runs. (Obviously!)

  • Typical storage hierarchy:

    • Main memory (RAM) for currently used data.

    • Disk for the main database (secondary storage).

    • Tapes for archiving older versions of the data (tertiary storage).


Disks

Disks

  • Secondary storage device of choice.

  • Main advantage over tapes: random access vs.sequential.

  • Data is stored and retrieved in units called disk blocks or pages.

  • Unlike RAM, time to retrieve a disk page varies depending upon location on disk.

    • Therefore, relative placement of pages on disk has major impact on DBMS performance!


Components of a disk

Tracks

Arm movement

Arm assembly

Components of a Disk

Spindle

Disk head

  • The platters spin (say, 90rps).

  • The arm assembly is moved in or out to position a head on a desired track. Tracks under heads make a cylinder(imaginary!).

Sector

Platters

  • Only one head reads/writes at any one time.

  • Block size is a multiple of sector size (which is fixed).


Accessing a disk page

Accessing a Disk Page

  • Time to access (read/write) a disk block:

    • seek time (moving arms to position disk head on track)

    • rotational delay (waiting for block to rotate under head)

    • transfer time (actually moving data to/from disk surface)

  • Seek time and rotational delay dominate.

    • Seek time varies from about 1 to 20msec

    • Rotational delay varies from 0 to 10msec

    • Transfer rate is about 1msec per 4KB page

  • Key to lower I/O cost: reduce seek/rotation delays! Hardware vs. software solutions?


Arranging pages on disk

Arranging Pages on Disk

  • `Next’ block concept:

    • blocks on same track, followed by

    • blocks on same cylinder, followed by

    • blocks on adjacent cylinder

  • Blocks in a file should be arranged sequentially on disk (by `next’), to minimize seek and rotational delay.

  • For a sequential scan, pre-fetchingseveral pages at a time is a big win!


Review of database systems concepts chapters 1 2 3 4 5 8 9 in r

RAID

  • Disk Array: Arrangement of several disks that gives abstraction of a single, large disk.

  • Goals: Increase performance and reliability.

  • Two main techniques:

    • Data striping: Data is partitioned; size of a partition is called the striping unit. Partitions are distributed over several disks.

    • Redundancy: More disks => more failures. Redundant information allows reconstruction of data if a disk fails.


Raid levels

RAID Levels

  • Level 0: No redundancy

  • Level 1: Mirrored (two identical copies)

    • Each disk has a mirror image (check disk)

    • Parallel reads, a write involves two disks.

    • Maximum transfer rate = transfer rate of one disk

  • Level 0+1: Striping and Mirroring

    • Parallel reads, a write involves two disks.

    • Maximum transfer rate = aggregate bandwidth


Raid levels contd

RAID Levels (Contd.)

  • Level 3: Bit-Interleaved Parity

    • Striping Unit: One bit. One check disk.

    • Each read and write request involves all disks; disk array can process one request at a time.

  • Level 4: Block-Interleaved Parity

    • Striping Unit: One disk block. One check disk.

    • Parallel reads possible for small requests, large requests can utilize full bandwidth

    • Writes involve modified block and check disk

  • Level 5: Block-Interleaved Distributed Parity

    • Similar to RAID Level 4, but parity blocks are distributed over all disks


Disk space management

Disk Space Management

  • Lowest layer of DBMS software manages space on disk.

  • Higher levels call upon this layer to:

    • allocate/de-allocate a page

    • read/write a page

  • Request for a sequence of pages must be satisfied by allocating the pages sequentially on disk! Higher levels don’t need to know how this is done, or how free space is managed.


Disk space management popular methods

Disk Space ManagementPopular Methods:

  • Linked lists – free blocks list (First fit, Best fit, etc.)

  • Inodes in Unix – Tree of free block addresses

  • Bit map

  • Note, sometimes access to logical block i, involves several physical blocks (e.g. Unix)!


Buffer management in a dbms

DB

Buffer Management in a DBMS

Page Requests from Higher Levels

  • Data must be in RAM for DBMS to operate on it!

  • Table of <frame#, pageid> pairs is maintained.

BUFFER POOL

disk page

free frame

MAIN MEMORY

DISK

choice of frame dictated

by replacement policy


When a page is requested

When a Page is Requested ...

  • If requested page is not in pool:

    • Choose a frame for replacement

    • If frame is dirty, write it to disk

    • Read requested page into chosen frame

  • Pin the page and return its address.

  • If requests can be predicted (e.g., sequential scans)

  • pages can be pre-fetchedseveral pages at a time!


More on buffer management

More on Buffer Management

  • Requestor of page must unpin it, and indicate whether page has been modified:

    • dirtybit is used for this.

  • Page in pool may be requested many times,

    • a pin count is used. A page is a candidate for replacement iff pin count = 0.

  • CC & recovery may entail additional I/O when a frame is chosen for replacement. (Write-Ahead Log protocol; more later.)


Buffer replacement policy

Buffer Replacement Policy

  • Frame is chosen for replacement by a replacement policy:

    • Least-recently-used (LRU), Clock, MRU etc.

  • Policy can have big impact on # of I/O’s; depends on the access pattern.

  • Sequential flooding: Nasty situation caused by LRU + repeated sequential scans.

    • # buffer frames < # pages in file means each page request causes an I/O. MRU much better in this situation (but not in all situations, of course).


Dbms vs os file system

DBMS vs. OS File System

OS does disk space & buffer mgmt: why not let OS manage these tasks?

  • Differences in OS support: portability issues

  • Some limitations, e.g., files can’t span disks.

  • Buffer management in DBMS requires ability to:

    • pin a page in buffer pool, force a page to disk (important for implementing CC & recovery),

    • adjust replacement policy, and pre-fetch pages based on access patterns in typical DB operations.


Record formats fixed length

Record Formats: Fixed Length

  • Information about field types same for all records in a file; stored in systemcatalogs.

  • Finding i’th field does not require scan of record.

F1

F2

F3

F4

L1

L2

L3

L4

Base address (B)

Address = B+L1+L2


Record formats variable length

4

$

$

$

$

Record Formats: Variable Length

  • Two alternative formats (# fields is fixed):

F1 F2 F3 F4

Fields Delimited by Special Symbols

Field

Count

F1 F2 F3 F4

Array of Field Offsets

  • Second offers direct access to i’th field, efficient storage

  • of nulls(special don’t know value); small directory overhead.


Page formats fixed length records

Page Formats: Fixed Length Records

  • Record id = <page id, slot #>. In first alternative, moving records for free space management changes rid; may not be acceptable.

Slot 1

Slot 1

Slot 2

Slot 2

Free

Space

. . .

. . .

Slot N

Slot N

Slot M

N

. . .

1

1

1

M

0

M ... 3 2 1

number

of records

number

of slots

PACKED

UNPACKED, BITMAP


Page formats variable length records

Page Formats: Variable Length Records

Rid = (i,N)

  • Can move records on page without changing rid; so, attractive for fixed-length records too.

Page i

Rid = (i,2)

Rid = (i,1)

N

Pointer

to start

of free

space

20

16

24

N . . . 2 1

# slots

SLOT DIRECTORY


Files of records

Files of Records

  • Page or block is OK when doing I/O, but higher levels of DBMS operate on records, and files of records.

  • FILE: A collection of pages, each containing a collection of records. Must support:

    • insert/delete/modify record

    • read a particular record (specified using record id)

    • scan all records (possibly with some conditions on the records to be retrieved)


Unordered heap files

Unordered (Heap) Files

  • Simplest file structure contains records in no particular order.

  • As file grows and shrinks, disk pages are allocated and de-allocated.

  • To support record level operations, we must:

    • keep track of the pages in a file

    • keep track of free space on pages

    • keep track of the records on a page

  • There are many alternatives for keeping track of this.


Heap file implemented as a list

Heap File Implemented as a List

  • The header page id and Heap file name must be stored someplace.

  • Each page contains 2 `pointers’ plus data.

Data

Page

Data

Page

Data

Page

Full Pages

Header

Page

Data

Page

Data

Page

Data

Page

Pages with

Free Space


Heap file using a page directory

Data

Page 1

Header

Page

Data

Page 2

Data

Page N

DIRECTORY

Heap File Using a Page Directory

  • The entry for a page can include the number of free bytes on the page.

  • The directory is a collection of pages; linked list implementation is just one alternative.

    • Much smaller than linked list of all HF pages!


Alternative file organizations

Alternative File Organizations

Many alternatives exist, each ideal for some situations, and not so good in others:

  • Heap (random order) files:Suitable when typical access is a file scan retrieving all records.

  • Sorted Files:Best if records must be retrieved in some order, or only a `range’ of records is needed.

  • Indexes: Data structures to organize records via trees or hashing.

    • Like sorted files, they speed up searches for a subset of records, based on values in certain (“search key”) fields

    • Updates are much faster than in sorted files.


Cost model for our analysis

Cost Model for Our Analysis

We ignore CPU costs, for simplicity:

  • B: The number of data pages

  • R: Number of records per page

  • D: (Average) time to read or write disk page

  • Measuring number of page I/O’s ignores gains of pre-fetching a sequence of pages; thus, even I/O cost is only approximated.

  • Average-case analysis; based on several simplistic assumptions.

  • Good enough to show the overall trends!


Assumptions in our analysis

Assumptions in Our Analysis

  • Heap Files:

    • Equality selection on key; exactly one match.

  • Sorted Files:

    • Files compacted after deletions.

  • Indexes:

    • Alt (2), (3): data entry size = 10% size of record

    • Hash: No overflow buckets.

      • 80% page occupancy => File size = 1.25 data size

    • Tree: 67% occupancy (this is typical).

      • Implies file size = 1.5 data size


Assumptions contd

Assumptions (contd.)

  • Scans:

    • Leaf levels of a tree-index are chained.

    • Index data-entries plus actual file scanned for unclustered indexes.

  • Range searches:

    • We use tree indexes to restrict the set of data records fetched, but ignore hash indexes.


Cost of operations

Cost of Operations

  • Several assumptions underlie these (rough) estimates!


Cost of operations1

Cost of Operations

  • Several assumptions underlie these (rough) estimates!


Indexes

Indexes

  • A heap file allows us to retrieve records:

    • By specifying the rid, or

    • By scanning all records sequentially.

  • Sometimes, we want to retrieve records by specifying the values in one or more fields, e.g,

    • Find all students in the “CS” department

    • Find all students with gpa > 3

  • Indexes are file structures that enable us to answer such value-based queries efficiently.


Indexes1

Indexes

  • An index on a file speeds up selections on the search key fields for the index.

    • Any subset of the fields of a relation can be the search key for an index on the relation.

    • Search key is not the same as key(minimal set of fields that uniquely identify a record in a relation).

  • An index contains a collection of data entries, and supports efficient retrieval of all data entries k*with a given key value k.

    • Given data entry k*, we can find record with key k in at most one disk I/O. (not really but ok for us…)


Alternatives for data entry k in index

Alternatives for Data Entry k*in Index

  • In a data entry k* we can store:

    • Data record with key value k, or

    • <k, rid of data record with search key value k>, or

    • <k, list of rids of data records with search key k>

  • Choice of alternative for data entries is orthogonal to the indexing technique used to locate data entries with a given key value k.

    • Examples of indexing techniques: B+ trees, hash-based structures

    • Typically, index contains auxiliary information that directs searches to the desired data entries


Alternatives for data entries contd

Alternatives for Data Entries (Contd.)

  • Alternative 1:

    • If this is used, index structure is a file organization for data records (instead of a Heap file or sorted file).

    • At most one index on a given collection of data records can use Alternative 1. (Otherwise, data records are duplicated, leading to redundant storage and potential inconsistency.)

    • If data records are very large, # of pages containing data entries is high. Implies size of auxiliary information in the index is also large, typically.


Alternatives for data entries contd1

Alternatives for Data Entries (Contd.)

  • Alternatives 2 and 3:

    • Data entries typically much smaller than data records. So, better than Alternative 1 with large data records, especially if search keys are small. (Portion of index structure used to direct search, which depends on size of data entries, is much smaller than with Alternative 1.)

    • Alternative 3 more compact than Alternative 2, but leads to variable sized data entries even if search keys are of fixed length.


Index classification

Index Classification

  • Primary vs. secondary: If search key contains primary key, then called primary index.

    • Unique index: Search key contains a candidate key.

  • Clustered vs. unclustered: If order of data records is the same as, or `close to’, order of data entries, then called clustered index.

    • Alternative 1 implies clustered; in practice, clustered also implies Alternative 1 (since sorted files are rare).

    • A file can be clustered on at most one search key.

    • Cost of retrieving data records through index varies greatly based on whether index is clustered or not!


Clustered vs unclustered index

Clustered vs. Unclustered Index

  • Suppose that Alternative (2) is used for data entries, and that the data records are stored in a Heap file.

    • To build clustered index, first sort the Heap file (with some free space on each page for future inserts).

    • Overflow pages may be needed for inserts. (Thus, order of data recs is `close to’, but not identical to, the sort order.)

Index entries

UNCLUSTERED

CLUSTERED

direct search for

data entries

Data entries

Data entries

(Index File)

(Data file)

Data Records

Data Records


Index classification cont

Index Classification – cont.

Examples of composite key

indexes using lexicographic order.

  • Composite Search Keys: Search on a combination of fields.

    • Equality query: Every field value is equal to a constant value. E.g. wrt <sal,age> index:

      • age=20 and sal =75

    • Range query: Some field value is not a constant. E.g.:

      • age =20; or age=20 and sal > 10

  • Data entries in index sorted by search key to support range queries.

    • Lexicographic order, or

    • Spatial order.

11,80

11

12

12,10

name

age

sal

12,20

12

13,75

bob

12

10

13

<age, sal>

cal

11

80

<age>

joe

12

20

10,12

sue

13

75

10

20

20,12

Data records

sorted by name

75,13

75

80,11

80

<sal, age>

<sal>

Data entries in index

sorted by <sal,age>

Data entries

sorted by <sal>


Classes of indexes

Classes of Indexes

  • Unique/non-unique – whether search key is unique/non-unique

  • Dense/Nondense – every/not every search key (every record for Unique key) in the data file has a corresponding pointer in the index.

  • Clustered/Nonclustered – order of index search key is equal/not equal to file order.

  • Primary/Secondary – search key equal/not equal to primary key. Also, the answer is a single/set of data file records.

  • Note: for Unique key – Dense index indexes every record, Non-dense does not index every record

  • Note: Non-dense index must be clustered!


Dense clustered index file

Dense Clustered Index File

  • Dense index — Index record appears for every search-key value in the file.


Example of sparse clustered index file

Example of Sparse Clustered Index File


Sparse index files

Sparse Index Files

  • Sparse Index: contains index records for only some search-key values.

    • Applicable when records are sequentially ordered on search-key

  • To locate a record with search-key value K we:

    • Find index record with largest search-key value < K

    • Search file sequentially starting at the record to which the index record points

  • Less space and less maintenance overhead for insertions and deletions.

  • Generally slower than dense index for locating records.

  • Good tradeoff: sparse index with an index entry for every block in file, corresponding to least search-key value in the block. (common for Hash indexes!)


Note on alternatives cost

Note on Alternatives cost

  • Alternative 1: entries are large (records), index is bigger (height), but avoids the +1!

  • Alternatives 2,3: entries are small (key+pointer), index is smaller, but need the +1 (m)


System catalogs

System Catalogs

  • For each index:

    • structure (e.g., B+ tree) and search key fields

  • For each relation:

    • name, file name, file structure (e.g., Heap file)

    • attribute name and type, for each attribute

    • index name, for each index

    • integrity constraints

  • For each view:

    • view name and definition

  • Plus statistics, authorization, buffer pool size, etc.

  • Catalogs are themselves stored as relations!


Attr cat attr name rel name type position

Attr_Cat(attr_name, rel_name, type, position)


Summary1

Summary

  • Disks provide cheap, non-volatile storage.

    • Random access, but cost depends on location of page on disk; important to arrange data sequentially to minimize seek and rotation delays.

  • Buffer manager brings pages into RAM.

    • Page stays in RAM until released by requestor.

    • Written to disk when frame chosen for replacement (which is sometime after requestor releases the page).

    • Choice of frame to replace based on replacement policy.

    • Tries to pre-fetch several pages at a time.


Summary contd

Summary (Contd.)

  • DBMS vs. OS File Support

    • DBMS needs features not found in many OS’s, e.g., forcing a page to disk, controlling the order of page writes to disk, files spanning disks, ability to control pre-fetching and page replacement policy based on predictable access patterns, etc.

  • Variable length record format with field offset directory offers support for direct access to i’th field and null values.

  • Slotted page format supports variable length records and allows records to move on page.


Summary contd1

Summary (Contd.)

  • File layer keeps track of pages in a file, and supports abstraction of a collection of records.

    • Pages with free space identified using linked list or directory structure (similar to how pages in file are kept track of).

  • Indexes support efficient retrieval of records based on the values in some fields.

  • Catalog relations store information about relations, indexes and views. (Information that is common to all records in a given collection.)


Summary cont

Summary (cont.)

  • Many alternative file organizations exist, each appropriate in some situation.

  • If selection queries are frequent, sorting the file or building an index is important.

    • Hash-based indexes only good for equality search.

    • Sorted files and tree-based indexes best for range search; also good for equality search. (Files rarely kept sorted in practice; B+ tree index is better.)

  • Index is a collection of data entries plus a way to quickly find entries with given key values.


Summary contd2

Summary (Contd.)

  • Data entries can be actual data records, <key, rid> pairs, or <key, rid-list> pairs.

    • Choice orthogonal to indexing technique used to locate data entries with a given key value.

  • Can have several indexes on a given file of data records, each with a different search key.

  • Indexes can be classified as clustered vs. unclustered, primary vs. secondary, and dense vs. sparse. Differences have important consequences for utility/performance.


Dbms concepts review

DBMS Concepts Review


Structure of a dbms

Query Optimization

and Execution

Relational Operators

Files and Access Methods

Buffer Management

Disk Space Management

DB

Structure of a DBMS

These layers

must consider

concurrency

control and

recovery

  • A typical DBMS has a layered architecture.

  • The figure does not show the concurrency control and recovery components.

  • This is one of several possible architectures; each system has its own variations.


Transaction an execution of a db program

Transaction: An Execution of a DB Program

  • Key concept is transaction, which is an atomicsequence of database actions (reads/writes).

  • Each transaction, executed completely, must leave the DB in a consistent stateif DB is consistent when the transaction begins.

    • Users can specify some simple integrity constraintson the data, and the DBMS will enforce these constraints.

    • Beyond this, the DBMS does not really understand the semantics of the data. (e.g., it does not understand how the interest on a bank account is computed).

    • Thus, ensuring that a transaction (run alone) preserves consistency is ultimately the user’s responsibility!


Concurrency control

Concurrency Control

  • Concurrent execution of user programs is essential for good DBMS performance.

    • Because disk accesses are frequent, and relatively slow, it is important to keep the cpu humming by working on several user programs concurrently.

  • Interleaving actions of different user programs can lead to inconsistency: e.g., check is cleared while account balance is being computed.

  • DBMS ensures such problems don’t arise: users can pretend they are using a single-user system.


Example

Example

  • Consider two transactions (Xacts):

T1:BEGIN A=A+100, B=B-100 END

T2:BEGIN A=1.06*A, B=1.06*B END

  • Intuitively, the first transaction is transferring $100 from B’s account to A’s account. The second is crediting both accounts with a 6% interest payment.

  • There is no guarantee that T1 will execute before T2 or vice-versa, if both are submitted together. However, the net effect must be equivalent to these two transactions running serially in some order.


Scheduling concurrent transactions

Scheduling Concurrent Transactions

  • DBMS ensures that execution of {T1, ... , Tn} is equivalent to some serial execution T1’ ... Tn’.

    • Before reading/writing an object, a transaction requests a lock on the object, and waits till the DBMS gives it the lock. All locks are released at the end of the transaction. (Strict 2PL locking protocol.)

    • Idea: If an action of Ti (say, writing X) affects Tj (which perhaps reads X), one of them, say Ti, will obtain the lock on X first and Tj is forced to wait until Ti completes; this effectively orders the transactions.

    • What if Tj already has a lock on Y and Ti later requests a lock on Y? (Deadlock!) Ti or Tj is abortedand restarted!


Ensuring atomicity

Ensuring Atomicity

  • DBMS ensures atomicity(all-or-nothing property) even if system crashes in the middle of a Xact.

  • Idea: Keep a log(history) of all actions carried out by the DBMS while executing a set of Xacts:

    • Before a change is made to the database, the corresponding log entry is forced to a safe location. (WAL protocol; OS support for this is often inadequate.)

    • After a crash, the effects of partially executed transactions are undone using the log. (Thanks to WAL, if log entry wasn’t saved before the crash, corresponding change was not applied to database!)


The log

The Log

  • The following actions are recorded in the log:

    • Ti writes an object: The old value and the new value.

      • Log record must go to disk beforethe changed page!

    • Ti commits/aborts: A log record indicating this action.

  • Log records chained together by Xact id, so it’s easy to undo a specific Xact (e.g., to resolve a deadlock).

  • Log is often duplexed and archived on “stable” storage.

  • All log related activities (and in fact, all CC related activities such as lock/unlock, dealing with deadlocks etc.) are handled transparently by the DBMS.


Review of database systems concepts chapters 1 2 3 4 5 8 9 in r

DBMS TYPES

  • Hierarchical – Pre-historic – IMS

  • Network – Historic –IDMS, ADABAS, lead to Object- Oriented

  • RELATIONAL- current – 95% of the market – Oracle, Informix, SQL/ Server, Progress, IBM DB2, etc.

  • Object- ORIENTED Current – lot of HuHa but very narrow market, mainly CAD AND Engineering – Objectivity, Versant, Jasmine

  • Object – Relational- Current / Future – SQL3, Informix UDO , Oracle-9, IBM DB2.


Review of database systems concepts chapters 1 2 3 4 5 8 9 in r

Database systens: a brief time line

EVENT:

PRE-1960S

1945-magnetic tapes developed (the first medium to allow searching).

1957- First commercial computer installed.

1959- McGee proposed the notion of generalized access to electronically stored data.

THE 60s

1961- The first generalized DBMS-GEs Integrated Data Store (IDS) designed by Bachman.

THE 70s – database technology experienced rapid growth.

1970- The relational model is developed by Ted Codd, an IBM research fellow.

1971- CODASYL Database Task Group Report.

1975- ACM Special Interest Group on Management of data organized first SIGMOD international conference.

1976- Entity- relationship (ER)model introduced by chen.

THE 80s- DBMSs developed for personal computers (DBASE, PARADOX, etc).

1983- ANSI/SPARC survey revealed>100 relational systems had been implemented by the beginning of the 80s.


Review of database systems concepts chapters 1 2 3 4 5 8 9 in r

Database systens: a brief time line

  • 1985- Preliminary SQL standard published. Business world influenced by “Fourth Generation Languages”.

  • *Trends in the ‘80s: extendable database systems:object- oriented DBMSs, client server architecture for distributed database.

  • The ’90s

  • * Demand for extending DBMS capabilities to meet new applications.

  • * Emergence of commercial object- oriented DBMSs.

  • * Demand for developed applications utilizing data from a variety of sources.

  • * Demand for exploiting massively parallel processors (MPPs).

  • Total victory by the relational mod

  • SQL 3

  • Object relational systems.

  • The ’00s

  • The emergence of XML and the integration of XML and Relational databases


Important historical systems 1970s

Important historical systems (1970s)

  • Ingres – U. of California – Berkeley [J1,J2]

  • System R – IBM research center – San Jose [J3]

  • Invented most of the concepts of relational systems of today. Had a tremendous impact on commercial systems and the SQL standard


Highlights of ingres

Highlights of Ingres

  • Process structure

    • 4 processes, communication between processes (limited address space)

  • Query language – QUEL – Relational Calc

  • Catalog (data dictionary) as Relations

    • Many Unix files

  • Access-types

    • Hashing & Isam, B-trees later

  • Query optimization

    • Decomposition – Inefficient!

  • Security

    • Query modification.


Review of database systems concepts chapters 1 2 3 4 5 8 9 in r

INGRES

TEMP

DOC

AUX

DATADIR

BIN

SOURCE

----

----

----

Binary code files

Source -C

Code files

System initialization

files

DATA

BASE

NAME

DATA

BASE

NAME

admin

----

----

----

Catalog relation

DBA relation

Other user relations


Highlights of system r

Highlights of system R

  • Query language – SEQUEL - >SQL

    • Full power, general + embedded language

  • Physical structure

    • Modular, B-tree, Clustered vs unclustered

  • Query optimization

    • Cost-based – very powerful

  • Security

    • Views, grant/revoke

  • Concurrency control/recovery

    • First general solutions


The compilation approach

The compilation approach

PL/ I source Program

Select names into $x

From EMP

Where Empno = $y

System R

Precompiler

(XPREP)

Modified PL/ I Program

Access module

PARSE

Code

Gen

OPT

call

Machine code

Ready to run

On RSS


Databases make these folks happy

Databases make these folks happy ...

  • End users and DBMS vendors

  • DB application programmers

    • E.g., smart webmasters

  • Database administrator (DBA)

    • Designs logical /physical schemas

    • Handles security and authorization

    • Data availability, crash recovery

    • Database tuning as needs evolve

Must understand how a DBMS works!


Summary2

Summary

  • DBMS used to maintain, query large datasets.

  • Benefits include recovery from system crashes, concurrent access, quick application development, data integrity and security.

  • Levels of abstraction give data independence.

  • A DBMS typically has a layered architecture.

  • DBAs hold responsible jobs and are well-paid! 

  • DBMS R&D is one of the broadest, most exciting areas in CS.


  • Login