The autoregressive contract: one forward pass per token
Inference is a single forward pass over frozen weights — no gradients, no optimizer state, no backward graph. But it is not one pass per sentence; it is one pass per token. Each emitted token appends to the context and the entire pass runs again, which is why generation is serial and why latency scales with output length, not prompt length.
No Backprop Tax
Training stores activations for every layer to replay them backward. Inference keeps only the KV cache; memory that would hold optimizer momentum and variance is freed entirely. A 7B model trains at ~112 GB but serves in ~14 GB.
The Serial Trap
Token N+1 depends on token N. No amount of parallel hardware changes the dependency chain — only speculative decoding breaks it, by guessing ahead and verifying in batch.
State Lives in the Cache
The model is stateless between requests but stateful within one: the KV cache is the conversation's working memory, and every architectural decision in serving flows from its growth curve.
A 200-token reply on a 2k-token prompt costs 14 GB of weight streaming per token plus 512 KB of new KV per token. The forward pass is O(N_ctx) per token because attention reads the whole cache.
- Streaming SSE responses exist precisely because E2E is serial — users read token 30 while token 80 generates.
- Stop-sequence detection and max-token caps are enforced token-by-token in the decode loop, engine-side.