Prefill requires compute-heavy matrix multiplications (high arithmetic intensity), whereas Decode requires memory-bandwidth-heavy weight streaming (low arithmetic intensity). Disaggregated serving places Prefill and Decode on physically separate GPU pools, transferring KV cache blocks across high-speed interconnects (NVLink/InfiniBand) during the handoff.
Optimal Hardware Matching
Prefill nodes can run on compute-dense cards (e.g., H100 with high batching), while Decode nodes run on memory-rich nodes or specialized bandwidth topologies.
Interference Isolation
Prevents heavy prompt prefills from degrading decode latency guarantees entirely, isolating noisy neighbors in multi-tenant fleets.
High-Speed KV Handoff
Requires ultra-low latency networking (RDMA / NVLink) to stream KV cache states from Prefill GPUs to Decode GPUs in under 10-20ms.
Transferring a 4,096-token KV cache for Llama-3-70B (~1.25 GB) over a 50 GB/s interconnect takes ~25 ms — a fraction of the compute time saved by isolating prefill.
REAL-WORLD PRODUCTION ENGINEERING
DistServe and Mooncake demonstrate 2-3x throughput gains for LLM serving clusters using Prefill/Decode disaggregation.
Used extensively in hyperscale cloud API endpoints where p99 latency SLOs are strictly enforced.