Course 4c: Inference Framework & KV Cache [Section 3.5]¶
Lab:
cache/manager.py· exp06 · systems deep-diveEvidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
Why PagedAttention assumptions break [Section 3.5.1]¶
Hybrid attention creates caches that differ per layer type: CSA entries every m=4 tokens, HCA entries every m′=128, indexer keys with their own dimension, sliding-window state, and buffered "not yet compressible" tail tokens. Uniform pages cannot serve policies this diverse, and kernels demand block alignment. V4 splits the cache into two pools:
State cache (fixed-size per request, treated like an SSM state): holds the SWA window and uncompressed tails pending compression.
Classical blocks spanning \(\mathrm{lcm}(m,m')=512\) tokens each, yielding exactly \(k_1=\mathrm{lcm}/m=128\) CSA entries and \(k_2=\mathrm{lcm}/m'=4\) HCA entries per block: integer tile alignment for both regimes simultaneously.
On-disk storage [Section 3.5.2]¶
Shared prefixes (agent system prompts!) should prefill once. Compressed entries persist directly; a prefix hit reuses everything up to the last complete block, recomputing only the incomplete tail. SWA entries are ~8× larger than compressed ones, so three strategies trade storage vs recompute:
| strategy | store | restore cost |
|---|---|---|
| Full SWA caching | all window entries | read last n_win, zero recompute, write-heavy |
| Periodic checkpointing | last n_win every p tokens | load nearest + recompute remainder |
| Zero SWA caching | nothing | recompute last n_win·L tokens using cached compressed entries |
Lab verification [MEASURED exp06]¶
Our manager reproduces the layout semantics at toy scale and proves the correctness invariant the paper implies but does not state as a test: restore(snapshot(k)) followed by any suffix yields byte-identical state to an uninterrupted run (for snapshots, disk persistence, and multi-session shared-prefix reuse alike). Measured on 128K-token prefixes: persist 4.9 MB in 0.06 s, reload ≈82 ms versus ≈1 s full prefill.