cs3223: stop at midterm

2025-10-13 17:18:01 +08:00
parent 0c4fd9b8f3
commit d41b75d722
2 changed files with 298 additions and 0 deletions
--- a/cs3223/midterm.pdf
+++ b/cs3223/midterm.pdf
--- a/cs3223/midterm.typ
+++ b/cs3223/midterm.typ
@@ -0,0 +1,298 @@
+#set page(paper: "a4", flipped: true, margin: 0.5cm, columns: 4)
+#set text(size: 8pt)
+#show heading: set block(spacing:0.6em)
+
+= Storage
+- Parts of disk
+  - Platter has 2 surfaces
+  - Surface has many tracks
+  - Each track is broken up into sectors
+  - Cylinder is the same tracks across all surfaces
+  - Block comprises of multiple sectors
+- *Disk Access Time*: $"Seek time" + "Rotational Delay" + "Transfer Time"$
+  - *Seek Time*: Move arms to position disk head
+  - *Rotational Delay*: $1/2 60/"RPM"$
+  - *Transfer time*(for n sectors): $n times "time for 1 revolution"/ "sectors per track"$
+    - $n$ is requested sectors on track
+
+- Access Order
+  + Contiguous Blocks within same track (same surface)
+  + Cylinder track within same cylinder
+  + next cylinder
+
+== Buffer Manager
+#image("buffer-manager.png")
+- Data stored in block sized pages called frames
+- Each frame maintains pin count(PC) and dirty flag
+=== Replacement Policies
+- Decide which unpinned page to replace
+- *LRU*: queue of pointers to frames with PC = 0
+- *clock*: LRU variant
+  - *Reference bit*: turns on when PC = 0
+  - Replace a page when ref bit off and PC = 0
+#image("clock-replacement-policy.png")
+
+== Files
+- Heap File Implementation
+  - Linked List
+    - 2 linked lists, 1 of free pages, 1 of data pages
+  - Page Directory Implementation
+    - Directory structure, 1 entry per page. 
+    - to insert, scan directory to find page with space to store record
+*Page Formats*
+- *RID* = (page id, slot number)
+- Fixed Length records
+  - Packed Organization: Store records in contiguous slots (requires swapping last item to deleted location during deletion)
+  - Unpacked organization: Use bit array to maintain free slots
+- *Variable Length Records*: Slotted page organization
+
+*Record Formats*
+- Fixed Length Records: Stored consecutively
+- Variable length Records
+  - Delimit fields with special symbols (F1, \$, F2 \$, F3)
+  - Array of field offsets ($o_1, o_2, o_3, F 1, F 2, F 3$)
+*Data Entry Formats*
+1. $k*$ is an actual data record (with search key value k)
+2. $k*$ is of the form *(k, rid)*
+3. $k*$ is of the form *(k, rid-list)* list of rids of data with key $k$
+
+= B+ Tree index
+- *Search key* is sequence of $k$ data attributes $k >= 1$
+- *Composite search key* if $k > 1$
+- *unique key* if search key contains _candidate_ key of table
+- index is stored as file
+- *Clustered index*: Ordering of data is same as data entries
+  - key is known as *clustering key*
+  - Format 1 index is clustered index (Assume format 2 and 3 to be unclustered)
+
+== Tree based Index
+- *root node* at level 0
+- Height of tree = no of levels of internal node
+- *Leaf nodes*
+  - level h, where h is height of tree
+- *internal nodes* store entries in form $(p_0, k_1, p_1, k_2, p_2, ..., p_n)$
+  - $k_1 < k_2 < ... < k_n$
+  - $p_i$ = disk page address
+- *Order* of index tree
+  - Each non-root node has $m in [d, 2d]$ entries
+  - Root node has $m in [1, 2d]$ entries
+- *Equality search*: At each _internal_ node $N$, find largest key $k_i$ in N, such that $k_i <= k$
+  - if $k_i$ exists, go subtree $p_i$, else $p_0$
+- *Range search*: First matching record, and traverse doubly linked list
+- *Min nodes at level* i is $2 times (d + 1)^(i-1), i >= 1$
+- *Max nodes at level* i is $(2d + 1)^(i)$
+
+=== Operations (Right sibling first, then left)
+=== Insertion
+ *Leaf node Overflow*
+  - Redistribute and then split
+  - *Split*: Create a new leaf $N$ with $d+1$ entries. Create a new index entry $(k, square.filled)$ where $k$ is smallest key in $N$
+  - *Redistribute*: If sibling is not full, take from it. If given right, update right's parent pointer, else current node's parent pointer
+ *Internal node Overflow*
+  - Node has $2d+1$ keys. 
+  - Push middle $(d+1)$-th key up to parent. 
+=== Deletion
+ *Leaf node*
+  - Redistribute then merge
+  - *Redistribution*
+    - Sibling must have $> d$ recordsto borrow
+    - Update parent pointers to right sibling's smallest key)
+  - *Merge*
+    - If sibling has $d$ entries, then merge
+    - Combine with sibling, and then remove parent node
+ *Internal Node Underflow*
+  - Let $N'$ be adjacent _sibling_ node of $N$ with $l, l > d$ entries
+  - Insert $(K, N' . p_i)$ into $N$, where $i$ is the leftmost(0) or rightmost entry(l)
+  - Replace $K$ in parent node with $N'.k_i$
+  - Remove $(p_i, k_i)$ entry from $N'$
+=== Bulk Loading
+ Sort entries by search keys. 
+ Load leaf pages with $2d$ entries
+ For each leaf page, insert index entry to rightmost parent page
+
+
+= Hash based Index
+== Static Hashing
+- Data stored in $N$ buckets, where hash function $h(dot)$ is used to id bucket
+  - record with key $k$ is inserted into $B_i, "where" i = h(k) mod N$
+- Bucket is primary data page with 0+ overflow data pages
+== Linear Hashing
+- Grows linearly by splitting buckets
+  - Systematic splitting: Bucket $B_i$ is split before $B_(i+1)$
+- Let $N_i = 2^i N_0$ be file size at beginning of round $i$
+- How to split bucket $B_i$
+  - Add bucket $B_j$ (split image of $B_i$)
+  - Redistribute entries in $B_i$ between $B_i$ and $B_j$
+  - `next++; if next == NLevel: (level++; next = 0)`
+=== Performance
+- Average: 1.2 IO for uniform data
+- Worst Case: Linear in number of entries
+
+== Extensible Hashing
+- Overflowed bucket is resolved by splitting overflowed bucket
+- No overflow pages, and order in which buckets are split is random
+- Directory of pointers to buckets, directory has $2^d$ entries
+  - $d$ is global depth of hashed file
+  - Each bucket maintains a local depth $l in [0, d]$
+  - Entries in a bucket of local depth $l$: same last $l$ bits
+=== Bucket Overflow
+- Number of directory entries could be more than number of buckets
+- Number of dir entries pointing to bucket = $2^(d-l)$
+- When bucket $B$ with depth $l$ overflows, 
+  - Increment local depth of $B$ to $l+1$
+  - Allocate split image $B'$
+  - Redistribute entries between $B$ and $B'$ using $(l+1)$th bit
+- if $l+1 > "global depth " d$
+  - Directory is doubled in size, , global depth to $d+1$
+  - New entries point to same bucket as corresponding entry
+- if $l+1 <= "global depth " d$
+  - Update dir entry corresponding to split bucket's directory entry to point to split image
+
+=== Bucket Deletion
+- $B_i$ & $B_j$(with same local depth $l$ and differ only in $l$th bit) can be merged if entries fit bin bucket
+  - $B_i$ is deallocated, $B_j$'s local depth decremented by 1. Directory entries that point to $B_i$ points to $B_j$
+=== Performance
+- At most 2 disk IOs for equality selection
+- Collisions: If they have same hashed value. 
+  - Need overflow pages if collisions exceed page capacity
+
+#colbreak()
+= Sorting
+== Notation
+#table(
+  columns: (auto, auto),
+  $|r|$, [pages for R],
+  $||r||$, [tuples in r],
+  $pi_L (R)$, [project column by list $L$ from $R$],
+  $pi_L^* (R)$, [project with duplicates],
+
+  $b_d$, [Data records that can fit on page],
+  $b_i$, [Data entries that can fit on page],
+  $b_r$, [RIDs that can fit on page],
+
+
+)
+== External Merge Sort
+- *File size*: $N$ pages
+- Memory pages available: $B$
+- *Pass 0*: Create  sorted runs
+  - Read and sort $B$ pages at a time
+- *Pass i*: Use $B-1$ pages for input, 1 for output, performing $B-1$-way merge sort
+- *Analysis*
+  - Sorted runs: $N_0 = ceil(N/B)$
+  - Total passes: $ceil(log_(B-1) (N_0))+1$
+  - Total I/O: $2 N (ceil(log_(B-1) (N_0))+1)$
+=== Optimized Merge Sort
+- Read and write in blocks of $b$ pages
+  - Allocate 1 Block for output
+  - Remaining memory for input: $floor(B/b)-1$ blocks
+- *Analysis*
+  - sorted runs: $N_0 = ceil(N/B)$
+  - Runs Merged at each pass $F = floor(B/b)-1$
+  - No of merge passes: $ceil(log_F (N_0))$(+1 for total)
+  - Total IO: $2 N (ceil(log_F (N_0))+1)$ 
+- *Sorting with B+ Trees*: IO Cost: $h$ + Scan of leaf pages + Heap access (If not covering index)
+
+== Projection
+=== Sort based approach
+- Extract attributes, Sort attributes, remove duplicates
+- *Analysis*
+  + Extract Attributes:  $|R|"(scan)" + |pi_L^*(R)| "(output)"$
+  + Sort Attributes: 
+    - $N_0 = ceil((|pi_L^*(R)|)/B)$
+    - Merging Passes: $log_(B-1) (N_0)$
+    - Total IO: $2 |pi_L^*(R)| (log_(B-1) (N_0)+1)$
+  + Remove Duplicates: $|pi_L^*(R)|$
+=== Optimized approach
+- Merge Split step 2 into Creating and Merging sorted runs, and merge into step 1 and 3 respectively
+- *Analysis* 
+  - *Step 1*
+    - $B-1$ pages for initial sorted run
+    - Sorted Runs: $N_0 = ceil((|pi^*_L (R)|) / (B-1))$
+    - Create sorted run = $|R| + |pi^*_L (R)|$
+  - *Step 2*
+    - Merging passes: $ceil(log_(B-1) (N_0))$
+    - Cost of merging: $2 |pi^*_L (R)| ceil(log_(B-1) (N_0))$
+    - Cost of merging excluding IO output: $(2 ceil(log_(B-1) (N_0))-1) |pi^*_L (R)| $
+
+=== Hash based approach
+- *Partitioning*
+  - Allocate 1 page for input, $B-1$ page for output. 
+  - Read 1 page at a time, for each tuple, create projection, hash($h$) to distribute to $B-1$ buffers
+  - Flush to disk when full. 
+- *Duplicate Elimination*
+  - For each partition $R_i$, create hash table, hash each tuple with hash function $h' != h$ to bucket $B_j$ if $t in.not B_j$
+- *Partition Overflow*: hash table for $pi^*_L (R_i)$ is larger than memory pages allocated for $pi_L (R)$
+
+- *Analysis*
+  - IO Cost (no partition overflow) : $|R| + 2|pi^*_L (R)|$
+    - Partitioning Phase: $|R| + |pi^*_L (R)|$
+    - Duplicate Elimination: $|pi^*_L (R)|$
+  - To Avoid partition overflows:
+    - $|R_i| = (|pi^*_L (R)|) / (B-1)$
+    - $B > "size of hash table", |R_i| times f$
+    - $B > sqrt(f times |pi^*_L (R)|)$
+
+= Selection
+- *Conjunct*: $1>=$ terms connected by $or$
+- *CNF predicate*: $1>=$ conjuncts connected by $and$
+- *Covered Conjunct* - predicate $p_i$ is covered conjunct if each attribute in $p_i$ is in key $K$ or include column of Index $I$
+  - $p = ("age" > 5) and ("height" = 180) and ("level" = 3)$
+  - $I_1 "key" = ("level", "weight", "height")$
+  - $p_c "wrt" I_1 = ("height" = 180) and ("level" = 3)$
+- *Primary Conjunct*
+  - $I$ matches $p$ if attributes in $p$ form prefix of $K$ and all comparison operators are equality except last
+  - $p_p$ is largest subset of conjuncts in $p$ such that $I$ matches $p_p$
+- $sigma_p (R)$: Select rows from $R$ that satisfy predicate $p$
+- Access Path: way of accessing data records / entries
+    - *Table Scan*: Scan all data pages (Cost: $|R|$)
+    - *Index Scan*: Scan index pages
+    - *Index Combination*: Combine from multiple index scans
+    - Scan/Combination can be followed by RID lookup to retrieve data
+- *Index only plan*: Query where it does not need to access any data tuples in $R$
+- *Covering Index*: $I$ is covering index if all of $R$s attribute in query is part of the key / include columns of $I$
+
+== B+ Trees
+- For Index Scan + RID Lookup, many matching RIDs could refer to same page
+  - Sort matching RIDs before performing lookup: Avoid retrieving same page
+
+=== Analysis
+#{
+let nin = [$N_"internal"$]
+let nle = [$N_"leaf"$]
+let nlo = [$N_"lookup"$]
+let nso = [$N_"sort"$]
+let nco = [$N_"combine"$]
+[
+Cost of index scan = $nin + #nle + nlo$
+- *#nin*: No of internal nodes accessed
+  - Height of B+ tree index
+  - $ "height(est)" = cases(
+  ceil(log_F (ceil( (||R||) / b_d))) &"if index is clustered", 
+  ceil(log_F (ceil( (||R||) / b_i))) &"otherwise", 
+) $
+- *#nlo*: Data pages accessed for RID lookups
+  - If $I$ is covering index for $sigma_p (R), nlo = 0$
+  - else $nlo = ||sigma_p_c (R)||$
+  - If matching RIDs are sorted before RID lookup
+    - $nlo = nso + min{||sigma_p_c (R)||, |R|}$
+- *#nso*: sorting matching RIDs 
+  - $nso = 0 "if" ceil( (||sigma_p_c (R)||) / b_r ) <= B$ (if RIDs can fit into $B$)
+  - $ nso = 2 ceil((||sigma_p_c (R)||) / b_r) ceil(log_(B-1) (N_0)), N_0 = ceil(ceil((||sigma_p_c (R)||) / b_r) / B) $
+    - Sorting with External Merge Sort
+    - #nso does'nt include read IO for pass 0 as its included in #nin and #nle
+    - #nso does'nt incldue write IO for final merging pass as RID is used for lookup
+- *#nle*: Leaf pages scanned for evaluating $sigma_p (R)$
+  - $nle = ceil((||sigma_p_p (R)||)/b_d)$ if clustered
+  - $nle = ceil((||sigma_p_p (R)||)/b_i)$ if unclustered
+- *Index Combination*
+- Cost = $nin^p + nle^p + nin^q + nle^q + nco + nlo$
+  - #nco: IO cost to compute join of $pi_p$ $pi_q$
+  - If $min{|pi_X_p (S_p)|, |pi_X_q (S_q)|} <= B$
+    - One of the join operands can fit in mem, then $nco = 0$
+== Hash based Index Scan
+- Cost: $N_"dir" + N_"bucket" + nlo$
+  - $N_"dir"$: no of directory pages accessed (1 if extensible hash, 0 otherwise)
+  - $N_"bucket"$: max no of index's primary/overflow pages accessed
+  - $nlo= nso + min{||sigma_p_c (R)||, |R|}$ if I is not covering index for $sigma_p (R)$
+]}