INFERENCE ENGINEERING

SCENE 16 / 24 · SERVING MIDDLEWARE, FRAMEWORKS & SCHEDULING

Never leave a slot empty

A request finishes? Swap the next one in mid-generation.

MAKESPAN · 15 STEPSNEW ARRIVALS JOIN MID-FLIGHT
R0R0R1R1R2R2R3R3R4R401234567891011

Static batching locks the group until the longest request finishes — padding wastes slots and arrivals wait outside. Continuous batching swaps finished requests for queued ones at iteration boundaries, which is why modern servers keep GPUs busy.

ARCHITECTURE & FORMULATIONModule 6: Serving Middleware, Frameworks & Scheduling

Continuous batching: ending the padding tax

Static batching waits for every request in a batch to finish, padding finished sequences with zero-tokens until the longest request completes. Continuous batching (or iteration-level scheduling) operates per iteration: when a request emits [EOS], its slot is immediately reclaimed and a new request from the queue enters mid-generation.

Iteration-Level Scheduling

Instead of scheduling batch-by-batch, the engine reschedules the batch slots before every single decode step. Finished requests leave; queued prompts step into prefill or decode.

Zero Padding Waste

Because sequences are decoupled, padding tokens drop to zero. Memory and compute that would have been wasted on pad tokens become real generated tokens.

Synergy with PagedAttention

Dynamic slot swapping is only possible when KV memory can be allocated and freed in non-contiguous chunks — continuous batching relies directly on PagedAttention underneath.

MATHEMATICAL FORMULATION · BATCH EFFICIENCY GAIN
efficiency = Σ len_i / (N_batch × max(len_i)) → continuous batching ≈ 1.0 efficiency

Static batching on a batch with lengths [10, 50, 200, 500] achieves (760 / 2000) = 38% compute efficiency (62% pad waste). Continuous batching achieves ~100% efficiency.

REAL-WORLD PRODUCTION ENGINEERING
  • Orca (OSDI '22) introduced iteration-level scheduling; vLLM, TGI, and TRT-LLM all adopted it as the standard engine core.
  • Metrics dashboards track 'slot utilization' (active requests / max batch size) to measure scheduler saturation.
← 15 · Force valid JSON
17 · Decode never freezes →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]