Continuous batching: ending the padding tax
Static batching waits for every request in a batch to finish, padding finished sequences with zero-tokens until the longest request completes. Continuous batching (or iteration-level scheduling) operates per iteration: when a request emits [EOS], its slot is immediately reclaimed and a new request from the queue enters mid-generation.
Iteration-Level Scheduling
Instead of scheduling batch-by-batch, the engine reschedules the batch slots before every single decode step. Finished requests leave; queued prompts step into prefill or decode.
Zero Padding Waste
Because sequences are decoupled, padding tokens drop to zero. Memory and compute that would have been wasted on pad tokens become real generated tokens.
Synergy with PagedAttention
Dynamic slot swapping is only possible when KV memory can be allocated and freed in non-contiguous chunks — continuous batching relies directly on PagedAttention underneath.
Static batching on a batch with lengths [10, 50, 200, 500] achieves (760 / 2000) = 38% compute efficiency (62% pad waste). Continuous batching achieves ~100% efficiency.
- Orca (OSDI '22) introduced iteration-level scheduling; vLLM, TGI, and TRT-LLM all adopted it as the standard engine core.
- Metrics dashboards track 'slot utilization' (active requests / max batch size) to measure scheduler saturation.