AI ENGVisual Encyclopedia

SCENE 17 / 24 · SERVING MIDDLEWARE, FRAMEWORKS & SCHEDULING

Decode never freezes

Slice big prompts into micro-chunks so active decodes keep ticking.

CHUNK SIZE2048 TOK × 15 SLICESDECODE STALL · 2765 → 2765
MONOLITHICprefill 30k tokens — every decode frozenCHUNKEDgold = prefill sliceteal = decode steps that kept ticking

Same total prefill work — but interleaved, so co-batched decodes never freeze for2.8s. Small TTFT cost buys large TPOT stability under load.

TECHNICAL BREAKDOWNModule 6: Serving Middleware, Frameworks & Scheduling

Chunked prefill: interleave prompt processing without decode stalls

A long prompt prefill (e.g., 32k tokens) takes hundreds of milliseconds or seconds of compute. In a naive scheduler, active decode requests freeze while the prefill completes, causing massive spikes in Inter-Token Latency (ITL). Chunked prefill breaks long prompts into micro-chunks (e.g., 512 tokens) and interleaves them alongside decode steps.

ITL Spike Elimination

Without chunking, a single 30k prompt prefill stalls all active streams for ~1-2 seconds. With chunking, the prefill is spread across 60 steps, keeping decode ITL bounded within SLOs.

Piggybacked Prefill

A chunk of prompt prefill is scheduled in the same batch step as ongoing decode tokens, saturating GPU compute units without starving memory-bound streams.

TTFT vs TPOT Tradeoff

Chunking slightly increases Time-To-First-Token (TTFT) for the long prompt, but protects TPOT and ITL for every other request on the GPU.

MATHEMATICAL FORMULATION · INTERLEAVE SCHEDULING
n_chunks = ⌈prompt_len / chunk_size⌉ · max_stall ≈ chunk_size / 1000 × prefill_ms_per_1k

A 30,720 token prompt with 2,048 chunk size creates 15 slices. Max decode stall drops from 2,764 ms down to ~184 ms per step, maintaining smooth token streaming.

REAL-WORLD PRODUCTION ENGINEERING
  • Sarathi-Serve and vLLM implement chunked prefill (also known as Sarathi scheduling) to guarantee strict p99 ITL limits under heavy prompt loads.
  • Chunk sizes are tuned based on model size and GPU memory bandwidth (e.g., 512 to 2048 tokens).
← 16 · Never leave a slot empty
18 · Two pools, two appetites →