SCENE 10 / 24 · THE RUNTIME & MEMORY MANAGEMENT LAYER
Half a megabyte per token
Attention remembers its past in the KV cache — memory becomes the budget.
SEQUENCE LENGTH2,048 TOKENS
BATCH 11.00 GB
BATCH 88.00 GB
BATCH 3232.00 GB
Formula: 2 × 32 layers × 32 kvHeads × 128 headDim × 2 bytes = 512.0 KB per token. A 32-way batch at full context carries a small model's worth of weights in cache alone.
ARCHITECTURE & FORMULATIONModule 4: The AI Runtime & Memory Management Layer
KV cache anatomy: the second model living in your VRAM
Attention must remember what it has already seen: Keys and Values for every layer, every head, every past token. The KV cache grows linearly with context and batch — and at long contexts it rivals the weights themselves. The entire 'KV economy' of serving is managing this second, growing allocation.
The Formula
bytes/token = 2 (K and V) × layers × kv_heads × head_dim × dtype_bytes. A 7B model at FP16 stores 512 KB per token — a 32k context is 16 GB per sequence.
GQA as Compression
Grouped-Query Attention shares K/V heads across query heads (Llama-2-70B: 8 KV heads for 64 queries), cutting the cache 8× with negligible quality cost.
The Budget Collision
Weights (fixed) + KV (growing) + activations must fit in HBM. Serving capacity is decided by how many concurrent KV caches fit — the true definition of 'batch slot'.
MATHEMATICAL FORMULATION · CACHE FOOTPRINT
KV_bytes = 2 × L × H_kv × d_head × dtype_bytes × N_ctx × B
7B (L=32, H_kv=32, d=128, fp16): 512 KB/token. At 32k context × 8 concurrent requests: 128 GB — twice the weights. This is why 70B models page, quantize, and evict caches.
REAL-WORLD PRODUCTION ENGINEERING
vLLM logs 'KV cache utilization' as its primary health metric; above ~90% preemption (recompute) begins.
MLA (DeepSeek) and cross-layer KV sharing push per-token cost down another order of magnitude.