INFERENCE ENGINEERING

SCENE 17 / 24 · SERVING MIDDLEWARE, FRAMEWORKS & SCHEDULING

Decode never freezes

Slice big prompts into micro-chunks so active decodes keep ticking.

CHUNK SIZE2048 TOK × 15 SLICESDECODE STALL · 2765 → 2765
MONOLITHICprefill 30k tokens — every decode frozenCHUNKEDgold = prefill sliceteal = decode steps that kept ticking

Same total prefill work — but interleaved, so co-batched decodes never freeze for2.8s. Small TTFT cost buys large TPOT stability under load.

ARCHITECTURE & FORMULATIONModule 6: Serving Middleware, Frameworks & Scheduling

Chunked prefill: interleave prompt processing without decode stalls

A long prompt prefill (e.g., 32k tokens) takes hundreds of milliseconds or seconds of compute. In a naive scheduler, active decode requests freeze while the prefill completes, causing massive spikes in Inter-Token Latency (ITL). Chunked prefill breaks long prompts into micro-chunks (e.g., 512 tokens) and interleaves them alongside decode steps.

ITL Spike Elimination

Without chunking, a single 30k prompt prefill stalls all active streams for ~1-2 seconds. With chunking, the prefill is spread across 60 steps, keeping decode ITL bounded within SLOs.

Piggybacked Prefill

A chunk of prompt prefill is scheduled in the same batch step as ongoing decode tokens, saturating GPU compute units without starving memory-bound streams.

TTFT vs TPOT Tradeoff

Chunking slightly increases Time-To-First-Token (TTFT) for the long prompt, but protects TPOT and ITL for every other request on the GPU.

MATHEMATICAL FORMULATION · INTERLEAVE SCHEDULING
n_chunks = ⌈prompt_len / chunk_size⌉ · max_stall ≈ chunk_size / 1000 × prefill_ms_per_1k

A 30,720 token prompt with 2,048 chunk size creates 15 slices. Max decode stall drops from 2,764 ms down to ~184 ms per step, maintaining smooth token streaming.

REAL-WORLD PRODUCTION ENGINEERING
  • Sarathi-Serve and vLLM implement chunked prefill (also known as Sarathi scheduling) to guarantee strict p99 ITL limits under heavy prompt loads.
  • Chunk sizes are tuned based on model size and GPU memory bandwidth (e.g., 512 to 2048 tokens).
← 16 · Never leave a slot empty
18 · Two pools, two appetites →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]