Distributed KV cache: offloading, streaming & tiered memory
As context windows expand to 100k+ tokens, KV cache footprint exceeds GPU VRAM. Tiered KV management offloads inactive or cold KV cache blocks to CPU RAM or NVMe storage over PCIe/RDMA, streaming them back to GPU VRAM only when needed, avoiding expensive recomputation.
Hierarchy of Memory
GPU HBM (fastest, tiny) → CPU RAM (fast, large) → NVMe SSD (slowest, massive). Managing KV caches across this hierarchy optimizes overall system cost.
Offload vs Recompute Decision
If fetching a cold KV cache from CPU RAM over PCIe takes longer than recomputing the prefill on GPU, recomputation is preferred.
RDMA Remote Caching
In multi-node clusters, a node with idle VRAM/RAM can host KV caches for other nodes using Remote Direct Memory Access (RDMA).
Transferring 10 GB of KV over PCIe Gen5 (64 GB/s) takes ~156 ms. Recomputing 20k tokens at 40ms/1k takes 800 ms. Fetching wins by 5x.
- FlexGen and DeepSpeed-Inference pioneer tiered KV offloading to serve long-context models on single consumer GPUs.
- vLLM's multi-tier cache manager coordinates page migration between GPU HBM and Host CPU RAM seamlessly.