AI ENGVisual Encyclopedia

SCENE 18 / 24 · SERVING MIDDLEWARE, FRAMEWORKS & SCHEDULING

Two pools, two appetites

Compute-hungry prefill and memory-bound decode run on separate GPUs.

KV PAYLOAD · 2.00 GBHANDOFF · 42.9 ms @ 50 GB/sPREFILL
PREFILL POOLcompute-hungry · batch bigDECODE POOLmemory-bound · stream weightsKV transfereach pool scales against its own SLO; the fabric bridges the handoff

Prefill rips through the prompt in parallel — it wants raw FLOPs.

TECHNICAL BREAKDOWNModule 6: Serving Middleware, Frameworks & Scheduling

Prefill/Decode disaggregation: physically isolating distinct workloads

Prefill requires compute-heavy matrix multiplications (high arithmetic intensity), whereas Decode requires memory-bandwidth-heavy weight streaming (low arithmetic intensity). Disaggregated serving places Prefill and Decode on physically separate GPU pools, transferring KV cache blocks across high-speed interconnects (NVLink/InfiniBand) during the handoff.

Optimal Hardware Matching

Prefill nodes can run on compute-dense cards (e.g., H100 with high batching), while Decode nodes run on memory-rich nodes or specialized bandwidth topologies.

Interference Isolation

Prevents heavy prompt prefills from degrading decode latency guarantees entirely, isolating noisy neighbors in multi-tenant fleets.

High-Speed KV Handoff

Requires ultra-low latency networking (RDMA / NVLink) to stream KV cache states from Prefill GPUs to Decode GPUs in under 10-20ms.

MATHEMATICAL FORMULATION · HANDOFF OVERHEAD
t_transfer = (2 × L × H_kv × d_head × dtype × N_prompt) / Network_BW_GBs

Transferring a 4,096-token KV cache for Llama-3-70B (~1.25 GB) over a 50 GB/s interconnect takes ~25 ms — a fraction of the compute time saved by isolating prefill.

REAL-WORLD PRODUCTION ENGINEERING
  • DistServe and Mooncake demonstrate 2-3x throughput gains for LLM serving clusters using Prefill/Decode disaggregation.
  • Used extensively in hyperscale cloud API endpoints where p99 latency SLOs are strictly enforced.