Inside the token loop: forward pass to appended token
Every generated token traverses the same pipeline: a full forward pass produces logits over the vocabulary; temperature rescales them; top-k and top-p truncate the candidate set; one sample is drawn; the token appends to the context. The loop is serial, and every stage between logits and sample is negligible next to the pass itself.
Logits Are Raw Scores
The final projection emits one score per vocabulary entry (~128k floats). Everything after is cheap vector arithmetic on a few dozen candidates.
Filter Order Matters
Temperature reshapes before truncation: at t→0 the distribution sharpens toward argmax regardless of top-k. Filters compose — penalties stack outside them.
The Append Is the State Change
Nothing else about the model changes between steps. Growth happens in the KV cache; that is the loop's only persistent mutation.
At t=1 the chain is the model's honest distribution; at t=0.1 it concentrates ~90% of mass on the top token; top-p=0.9 adapts the candidate count to distribution shape instead of fixing it.
- Sampling kernels are memory-trivial but latency-visible at batch 1: fused sampling kernels avoid a separate sync per step.
- Engines expose per-request seeds for reproducibility — the sampler is deterministic given the seed and the logits.