INFERENCE ENGINEERING

SCENE 22 / 24 · DISTRIBUTED INFERENCE & INFRASTRUCTURE SCALE

KV caches across nodes

Offload, stream and balance attention memory over the network fabric.

QUEUE DEPTH16,384 TOK × 8 REQ
HBM · HOT32.00 GBHOST MEM / NVMe · COLD32.00 GBRDMAfetch cold block over fabric · 1374 ms @ 25 GB/svs recompute from scratch · 1475 ms→ offload wins

Placement is the decision: streaming a cold KV block over the fabric beats recomputing it — until it doesn't. Balance keeps hot prefixes resident and long tails tiered.

ARCHITECTURE & FORMULATIONModule 7: Distributed Inference & Infrastructure Scale

Distributed KV cache: offloading, streaming & tiered memory

As context windows expand to 100k+ tokens, KV cache footprint exceeds GPU VRAM. Tiered KV management offloads inactive or cold KV cache blocks to CPU RAM or NVMe storage over PCIe/RDMA, streaming them back to GPU VRAM only when needed, avoiding expensive recomputation.

Hierarchy of Memory

GPU HBM (fastest, tiny) → CPU RAM (fast, large) → NVMe SSD (slowest, massive). Managing KV caches across this hierarchy optimizes overall system cost.

Offload vs Recompute Decision

If fetching a cold KV cache from CPU RAM over PCIe takes longer than recomputing the prefill on GPU, recomputation is preferred.

RDMA Remote Caching

In multi-node clusters, a node with idle VRAM/RAM can host KV caches for other nodes using Remote Direct Memory Access (RDMA).

MATHEMATICAL FORMULATION · OFFLOAD VS RECOMPUTE THRESHOLD
t_fetch = KV_bytes / PCIe_BW vs t_recompute = N_tokens / 1000 × prefill_ms_per_1k

Transferring 10 GB of KV over PCIe Gen5 (64 GB/s) takes ~156 ms. Recomputing 20k tokens at 40ms/1k takes 800 ms. Fetching wins by 5x.

REAL-WORLD PRODUCTION ENGINEERING
  • FlexGen and DeepSpeed-Inference pioneer tiered KV offloading to serve long-context models on single consumer GPUs.
  • vLLM's multi-tier cache manager coordinates page migration between GPU HBM and Host CPU RAM seamlessly.
← 21 · Route to a few experts
23 · The cold start tax →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]