Skip to content

KV Cache as a First-Class System (Exp06 + tests)

Paper Section 3.5.1–3.5.2 · Code: src/deepseek_v4_lab/cache/manager.py

Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger

Why this exists

Problem: heterogeneous entry classes (compressed CSA/HCA blocks, sliding window, pending tail) break uniform-page paged-cache assumptions.

Response: lcm-block layout so both compression regimes tile exactly; state buffer for tails; ring window; persist/reuse promoted to first-class operations.

New cost: allocator and kernel-alignment constraints; snapshot/restore correctness must be proven, not assumed.

Our experiment: byte-identical restore tests plus 128K-prefix persist/reload timings [MEASURED].

Exchange rate: BANDWIDTH buys hidden wait time; spends layout discipline and correctness proof obligations.

Design implemented [IMPLEMENTED]

Three entry classes with distinct rules, mirroring the paper's layout:

class granularity update rule eviction
completed blocks lcm(m, m′)=128 tokens → 32 CSA + 1 HCA entries pooled on block completion oldest-first
tail buffer pending tokens < block boundary grows until compressible never (state)
sliding window last n_win raw entries ring buffer implicit

Correctness invariants (tests, not perf claims)

  • test_snapshot_restore_equivalence: restore(snapshot(k)) + same suffix ⇒ byte-identical state to uninterrupted run
  • test_persist_roundtrip and test_shared_prefix_reuse: two sessions restoring one persisted prefix then diverging each match their twins
  • test_block_alignment_lcm: k₁=lcm/m, k₂=lcm/m′ per block [PAPER Section 3.5.1]

Toy-scale result. Direction is informative; production magnitude is not.

Measured (Exp06)

128K-token prefix: full prefill ≈ 1.04 s; persist 4.9 MB in 0.06 s; load ≈ 82 ms. Reuse turns O(prefix) prefill into O(suffix) append after a cheap load, the shared-prefix economics the paper builds on-disk caching for.

Limitations

  • Uniform pooling weights (content-dependent pooling is a model concern, not a cache concern).
  • No paged allocator / kernel alignment constraints (paper's second obstacle): our blocks are exact lcm tiles by construction.