KV cache anatomy: the second model living in your VRAM
Attention must remember what it has already seen: Keys and Values for every layer, every head, every past token. The KV cache grows linearly with context and batch — and at long contexts it rivals the weights themselves. The entire 'KV economy' of serving is managing this second, growing allocation.
The Formula
bytes/token = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes. A 7B model at FP16 stores 512 KB per token — a 32k context is 16 GB per sequence.
GQA as Compression
Grouped-Query Attention shares K/V heads across query heads (Llama-2-70B: 8 KV heads for 64 queries), cutting the cache 8× with negligible quality cost.
The Budget Collision
Weights (fixed) + KV (growing) + activations must fit in HBM. Serving capacity is decided by how many concurrent KV caches fit — the true definition of 'batch slot'.
7B (L=32, H_kv=32, d=128, fp16): 512 KB/token. At 32k context × 8 concurrent requests: 128 GB — twice the weights. This is why 70B models page, quantize, and evict caches.
- vLLM logs 'KV cache utilization' as its primary health metric; above ~90% preemption (recompute) begins.
- MLA (DeepSeek) and cross-layer KV sharing push per-token cost down another order of magnitude.