SCENE 01 / 24 · FOUNDATIONS OF INFERENCE & EXECUTION
One token at a time
An LLM never writes a sentence — it predicts the next token, then does it again.
STEP 0 / 16KV CACHE · 512.0 KB
Each chip is one forward pass over the whole model. Past tokens join the input every step — that is why generation is serial and why the KV cache grows without asking.
ARCHITECTURE & FORMULATIONModule 1: Foundations of Inference & Execution
The autoregressive contract: one forward pass per token
Inference is a single forward pass over frozen weights — no gradients, no optimizer state, no backward graph. But it is not one pass per sentence; it is one pass per token. Each emitted token appends to the context and the entire pass runs again, which is why generation is serial and why latency scales with output length, not prompt length.
No Backprop Tax
Training stores activations for every layer to replay them backward. Inference keeps only the KV cache; memory that would hold optimizer momentum and variance is freed entirely. A 7B model trains at ~112 GB but serves in ~14 GB.
The Serial Trap
Token N+1 depends on token N. No amount of parallel hardware changes the dependency chain — only speculative decoding breaks it, by guessing ahead and verifying in batch.
State Lives in the Cache
The model is stateless between requests but stateful within one: the KV cache is the conversation's working memory, and every architectural decision in serving flows from its growth curve.
A 200-token reply on a 2k-token prompt costs 14 GB of weight streaming per token plus 512 KB of new KV per token. The forward pass is O(N_ctx) per token because attention reads the whole cache.
REAL-WORLD PRODUCTION ENGINEERING
Streaming SSE responses exist precisely because E2E is serial — users read token 30 while token 80 generates.
Stop-sequence detection and max-token caps are enforced token-by-token in the decode loop, engine-side.