299 lines
12 KiB
Typst
299 lines
12 KiB
Typst
#set page(paper: "a4", flipped: true, margin: 0.5cm, columns: 4)
|
|
#set text(size: 8pt)
|
|
#show heading: set block(spacing:0.6em)
|
|
|
|
= Storage
|
|
- Parts of disk
|
|
- Platter has 2 surfaces
|
|
- Surface has many tracks
|
|
- Each track is broken up into sectors
|
|
- Cylinder is the same tracks across all surfaces
|
|
- Block comprises of multiple sectors
|
|
- *Disk Access Time*: $"Seek time" + "Rotational Delay" + "Transfer Time"$
|
|
- *Seek Time*: Move arms to position disk head
|
|
- *Rotational Delay*: $1/2 60/"RPM"$
|
|
- *Transfer time*(for n sectors): $n times "time for 1 revolution"/ "sectors per track"$
|
|
- $n$ is requested sectors on track
|
|
|
|
- Access Order
|
|
+ Contiguous Blocks within same track (same surface)
|
|
+ Cylinder track within same cylinder
|
|
+ next cylinder
|
|
|
|
== Buffer Manager
|
|
#image("buffer-manager.png")
|
|
- Data stored in block sized pages called frames
|
|
- Each frame maintains pin count(PC) and dirty flag
|
|
=== Replacement Policies
|
|
- Decide which unpinned page to replace
|
|
- *LRU*: queue of pointers to frames with PC = 0
|
|
- *clock*: LRU variant
|
|
- *Reference bit*: turns on when PC = 0
|
|
- Replace a page when ref bit off and PC = 0
|
|
#image("clock-replacement-policy.png")
|
|
|
|
== Files
|
|
- Heap File Implementation
|
|
- Linked List
|
|
- 2 linked lists, 1 of free pages, 1 of data pages
|
|
- Page Directory Implementation
|
|
- Directory structure, 1 entry per page.
|
|
- to insert, scan directory to find page with space to store record
|
|
*Page Formats*
|
|
- *RID* = (page id, slot number)
|
|
- Fixed Length records
|
|
- Packed Organization: Store records in contiguous slots (requires swapping last item to deleted location during deletion)
|
|
- Unpacked organization: Use bit array to maintain free slots
|
|
- *Variable Length Records*: Slotted page organization
|
|
|
|
*Record Formats*
|
|
- Fixed Length Records: Stored consecutively
|
|
- Variable length Records
|
|
- Delimit fields with special symbols (F1, \$, F2 \$, F3)
|
|
- Array of field offsets ($o_1, o_2, o_3, F 1, F 2, F 3$)
|
|
*Data Entry Formats*
|
|
1. $k*$ is an actual data record (with search key value k)
|
|
2. $k*$ is of the form *(k, rid)*
|
|
3. $k*$ is of the form *(k, rid-list)* list of rids of data with key $k$
|
|
|
|
= B+ Tree index
|
|
- *Search key* is sequence of $k$ data attributes $k >= 1$
|
|
- *Composite search key* if $k > 1$
|
|
- *unique key* if search key contains _candidate_ key of table
|
|
- index is stored as file
|
|
- *Clustered index*: Ordering of data is same as data entries
|
|
- key is known as *clustering key*
|
|
- Format 1 index is clustered index (Assume format 2 and 3 to be unclustered)
|
|
|
|
== Tree based Index
|
|
- *root node* at level 0
|
|
- Height of tree = no of levels of internal node
|
|
- *Leaf nodes*
|
|
- level h, where h is height of tree
|
|
- *internal nodes* store entries in form $(p_0, k_1, p_1, k_2, p_2, ..., p_n)$
|
|
- $k_1 < k_2 < ... < k_n$
|
|
- $p_i$ = disk page address
|
|
- *Order* of index tree
|
|
- Each non-root node has $m in [d, 2d]$ entries
|
|
- Root node has $m in [1, 2d]$ entries
|
|
- *Equality search*: At each _internal_ node $N$, find largest key $k_i$ in N, such that $k_i <= k$
|
|
- if $k_i$ exists, go subtree $p_i$, else $p_0$
|
|
- *Range search*: First matching record, and traverse doubly linked list
|
|
- *Min nodes at level* i is $2 times (d + 1)^(i-1), i >= 1$
|
|
- *Max nodes at level* i is $(2d + 1)^(i)$
|
|
|
|
=== Operations (Right sibling first, then left)
|
|
=== Insertion
|
|
+ *Leaf node Overflow*
|
|
- Redistribute and then split
|
|
- *Split*: Create a new leaf $N$ with $d+1$ entries. Create a new index entry $(k, square.filled)$ where $k$ is smallest key in $N$
|
|
- *Redistribute*: If sibling is not full, take from it. If given right, update right's parent pointer, else current node's parent pointer
|
|
+ *Internal node Overflow*
|
|
- Node has $2d+1$ keys.
|
|
- Push middle $(d+1)$-th key up to parent.
|
|
=== Deletion
|
|
+ *Leaf node*
|
|
- Redistribute then merge
|
|
- *Redistribution*
|
|
- Sibling must have $> d$ recordsto borrow
|
|
- Update parent pointers to right sibling's smallest key)
|
|
- *Merge*
|
|
- If sibling has $d$ entries, then merge
|
|
- Combine with sibling, and then remove parent node
|
|
+ *Internal Node Underflow*
|
|
- Let $N'$ be adjacent _sibling_ node of $N$ with $l, l > d$ entries
|
|
- Insert $(K, N' . p_i)$ into $N$, where $i$ is the leftmost(0) or rightmost entry(l)
|
|
- Replace $K$ in parent node with $N'.k_i$
|
|
- Remove $(p_i, k_i)$ entry from $N'$
|
|
=== Bulk Loading
|
|
+ Sort entries by search keys.
|
|
+ Load leaf pages with $2d$ entries
|
|
+ For each leaf page, insert index entry to rightmost parent page
|
|
|
|
|
|
= Hash based Index
|
|
== Static Hashing
|
|
- Data stored in $N$ buckets, where hash function $h(dot)$ is used to id bucket
|
|
- record with key $k$ is inserted into $B_i, "where" i = h(k) mod N$
|
|
- Bucket is primary data page with 0+ overflow data pages
|
|
== Linear Hashing
|
|
- Grows linearly by splitting buckets
|
|
- Systematic splitting: Bucket $B_i$ is split before $B_(i+1)$
|
|
- Let $N_i = 2^i N_0$ be file size at beginning of round $i$
|
|
- How to split bucket $B_i$
|
|
- Add bucket $B_j$ (split image of $B_i$)
|
|
- Redistribute entries in $B_i$ between $B_i$ and $B_j$
|
|
- `next++; if next == NLevel: (level++; next = 0)`
|
|
=== Performance
|
|
- Average: 1.2 IO for uniform data
|
|
- Worst Case: Linear in number of entries
|
|
|
|
== Extensible Hashing
|
|
- Overflowed bucket is resolved by splitting overflowed bucket
|
|
- No overflow pages, and order in which buckets are split is random
|
|
- Directory of pointers to buckets, directory has $2^d$ entries
|
|
- $d$ is global depth of hashed file
|
|
- Each bucket maintains a local depth $l in [0, d]$
|
|
- Entries in a bucket of local depth $l$: same last $l$ bits
|
|
=== Bucket Overflow
|
|
- Number of directory entries could be more than number of buckets
|
|
- Number of dir entries pointing to bucket = $2^(d-l)$
|
|
- When bucket $B$ with depth $l$ overflows,
|
|
- Increment local depth of $B$ to $l+1$
|
|
- Allocate split image $B'$
|
|
- Redistribute entries between $B$ and $B'$ using $(l+1)$th bit
|
|
- if $l+1 > "global depth " d$
|
|
- Directory is doubled in size, , global depth to $d+1$
|
|
- New entries point to same bucket as corresponding entry
|
|
- if $l+1 <= "global depth " d$
|
|
- Update dir entry corresponding to split bucket's directory entry to point to split image
|
|
|
|
=== Bucket Deletion
|
|
- $B_i$ & $B_j$(with same local depth $l$ and differ only in $l$th bit) can be merged if entries fit bin bucket
|
|
- $B_i$ is deallocated, $B_j$'s local depth decremented by 1. Directory entries that point to $B_i$ points to $B_j$
|
|
=== Performance
|
|
- At most 2 disk IOs for equality selection
|
|
- Collisions: If they have same hashed value.
|
|
- Need overflow pages if collisions exceed page capacity
|
|
|
|
#colbreak()
|
|
= Sorting
|
|
== Notation
|
|
#table(
|
|
columns: (auto, auto),
|
|
$|r|$, [pages for R],
|
|
$||r||$, [tuples in r],
|
|
$pi_L (R)$, [project column by list $L$ from $R$],
|
|
$pi_L^* (R)$, [project with duplicates],
|
|
|
|
$b_d$, [Data records that can fit on page],
|
|
$b_i$, [Data entries that can fit on page],
|
|
$b_r$, [RIDs that can fit on page],
|
|
|
|
|
|
)
|
|
== External Merge Sort
|
|
- *File size*: $N$ pages
|
|
- Memory pages available: $B$
|
|
- *Pass 0*: Create sorted runs
|
|
- Read and sort $B$ pages at a time
|
|
- *Pass i*: Use $B-1$ pages for input, 1 for output, performing $B-1$-way merge sort
|
|
- *Analysis*
|
|
- Sorted runs: $N_0 = ceil(N/B)$
|
|
- Total passes: $ceil(log_(B-1) (N_0))+1$
|
|
- Total I/O: $2 N (ceil(log_(B-1) (N_0))+1)$
|
|
=== Optimized Merge Sort
|
|
- Read and write in blocks of $b$ pages
|
|
- Allocate 1 Block for output
|
|
- Remaining memory for input: $floor(B/b)-1$ blocks
|
|
- *Analysis*
|
|
- sorted runs: $N_0 = ceil(N/B)$
|
|
- Runs Merged at each pass $F = floor(B/b)-1$
|
|
- No of merge passes: $ceil(log_F (N_0))$(+1 for total)
|
|
- Total IO: $2 N (ceil(log_F (N_0))+1)$
|
|
- *Sorting with B+ Trees*: IO Cost: $h$ + Scan of leaf pages + Heap access (If not covering index)
|
|
|
|
== Projection
|
|
=== Sort based approach
|
|
- Extract attributes, Sort attributes, remove duplicates
|
|
- *Analysis*
|
|
+ Extract Attributes: $|R|"(scan)" + |pi_L^*(R)| "(output)"$
|
|
+ Sort Attributes:
|
|
- $N_0 = ceil((|pi_L^*(R)|)/B)$
|
|
- Merging Passes: $log_(B-1) (N_0)$
|
|
- Total IO: $2 |pi_L^*(R)| (log_(B-1) (N_0)+1)$
|
|
+ Remove Duplicates: $|pi_L^*(R)|$
|
|
=== Optimized approach
|
|
- Merge Split step 2 into Creating and Merging sorted runs, and merge into step 1 and 3 respectively
|
|
- *Analysis*
|
|
- *Step 1*
|
|
- $B-1$ pages for initial sorted run
|
|
- Sorted Runs: $N_0 = ceil((|pi^*_L (R)|) / (B-1))$
|
|
- Create sorted run = $|R| + |pi^*_L (R)|$
|
|
- *Step 2*
|
|
- Merging passes: $ceil(log_(B-1) (N_0))$
|
|
- Cost of merging: $2 |pi^*_L (R)| ceil(log_(B-1) (N_0))$
|
|
- Cost of merging excluding IO output: $(2 ceil(log_(B-1) (N_0))-1) |pi^*_L (R)| $
|
|
|
|
=== Hash based approach
|
|
- *Partitioning*
|
|
- Allocate 1 page for input, $B-1$ page for output.
|
|
- Read 1 page at a time, for each tuple, create projection, hash($h$) to distribute to $B-1$ buffers
|
|
- Flush to disk when full.
|
|
- *Duplicate Elimination*
|
|
- For each partition $R_i$, create hash table, hash each tuple with hash function $h' != h$ to bucket $B_j$ if $t in.not B_j$
|
|
- *Partition Overflow*: hash table for $pi^*_L (R_i)$ is larger than memory pages allocated for $pi_L (R)$
|
|
|
|
- *Analysis*
|
|
- IO Cost (no partition overflow) : $|R| + 2|pi^*_L (R)|$
|
|
- Partitioning Phase: $|R| + |pi^*_L (R)|$
|
|
- Duplicate Elimination: $|pi^*_L (R)|$
|
|
- To Avoid partition overflows:
|
|
- $|R_i| = (|pi^*_L (R)|) / (B-1)$
|
|
- $B > "size of hash table", |R_i| times f$
|
|
- $B > sqrt(f times |pi^*_L (R)|)$
|
|
|
|
= Selection
|
|
- *Conjunct*: $1>=$ terms connected by $or$
|
|
- *CNF predicate*: $1>=$ conjuncts connected by $and$
|
|
- *Covered Conjunct* - predicate $p_i$ is covered conjunct if each attribute in $p_i$ is in key $K$ or include column of Index $I$
|
|
- $p = ("age" > 5) and ("height" = 180) and ("level" = 3)$
|
|
- $I_1 "key" = ("level", "weight", "height")$
|
|
- $p_c "wrt" I_1 = ("height" = 180) and ("level" = 3)$
|
|
- *Primary Conjunct*
|
|
- $I$ matches $p$ if attributes in $p$ form prefix of $K$ and all comparison operators are equality except last
|
|
- $p_p$ is largest subset of conjuncts in $p$ such that $I$ matches $p_p$
|
|
- $sigma_p (R)$: Select rows from $R$ that satisfy predicate $p$
|
|
- Access Path: way of accessing data records / entries
|
|
- *Table Scan*: Scan all data pages (Cost: $|R|$)
|
|
- *Index Scan*: Scan index pages
|
|
- *Index Combination*: Combine from multiple index scans
|
|
- Scan/Combination can be followed by RID lookup to retrieve data
|
|
- *Index only plan*: Query where it does not need to access any data tuples in $R$
|
|
- *Covering Index*: $I$ is covering index if all of $R$s attribute in query is part of the key / include columns of $I$
|
|
|
|
== B+ Trees
|
|
- For Index Scan + RID Lookup, many matching RIDs could refer to same page
|
|
- Sort matching RIDs before performing lookup: Avoid retrieving same page
|
|
|
|
=== Analysis
|
|
#{
|
|
let nin = [$N_"internal"$]
|
|
let nle = [$N_"leaf"$]
|
|
let nlo = [$N_"lookup"$]
|
|
let nso = [$N_"sort"$]
|
|
let nco = [$N_"combine"$]
|
|
[
|
|
Cost of index scan = $nin + #nle + nlo$
|
|
- *#nin*: No of internal nodes accessed
|
|
- Height of B+ tree index
|
|
- $ "height(est)" = cases(
|
|
ceil(log_F (ceil( (||R||) / b_d))) &"if index is clustered",
|
|
ceil(log_F (ceil( (||R||) / b_i))) &"otherwise",
|
|
) $
|
|
- *#nlo*: Data pages accessed for RID lookups
|
|
- If $I$ is covering index for $sigma_p (R), nlo = 0$
|
|
- else $nlo = ||sigma_p_c (R)||$
|
|
- If matching RIDs are sorted before RID lookup
|
|
- $nlo = nso + min{||sigma_p_c (R)||, |R|}$
|
|
- *#nso*: sorting matching RIDs
|
|
- $nso = 0 "if" ceil( (||sigma_p_c (R)||) / b_r ) <= B$ (if RIDs can fit into $B$)
|
|
- $ nso = 2 ceil((||sigma_p_c (R)||) / b_r) ceil(log_(B-1) (N_0)), N_0 = ceil(ceil((||sigma_p_c (R)||) / b_r) / B) $
|
|
- Sorting with External Merge Sort
|
|
- #nso does'nt include read IO for pass 0 as its included in #nin and #nle
|
|
- #nso does'nt incldue write IO for final merging pass as RID is used for lookup
|
|
- *#nle*: Leaf pages scanned for evaluating $sigma_p (R)$
|
|
- $nle = ceil((||sigma_p_p (R)||)/b_d)$ if clustered
|
|
- $nle = ceil((||sigma_p_p (R)||)/b_i)$ if unclustered
|
|
- *Index Combination*
|
|
- Cost = $nin^p + nle^p + nin^q + nle^q + nco + nlo$
|
|
- #nco: IO cost to compute join of $pi_p$ $pi_q$
|
|
- If $min{|pi_X_p (S_p)|, |pi_X_q (S_q)|} <= B$
|
|
- One of the join operands can fit in mem, then $nco = 0$
|
|
== Hash based Index Scan
|
|
- Cost: $N_"dir" + N_"bucket" + nlo$
|
|
- $N_"dir"$: no of directory pages accessed (1 if extensible hash, 0 otherwise)
|
|
- $N_"bucket"$: max no of index's primary/overflow pages accessed
|
|
- $nlo= nso + min{||sigma_p_c (R)||, |R|}$ if I is not covering index for $sigma_p (R)$
|
|
]}
|