AI ENGVisual Encyclopedia

SCENE 13 / 24 · INTERACTION, SAMPLING & CONSTRAINED GENERATION

Logits in, token out

Temperature, top-k and top-p sculpt probabilities before the dice roll.

STEP · 1 / 6
FORWARD PASSLOGITSTEMPERATURETOP-K / TOP-PSAMPLEAPPEND → LOOPnext token joins the input

One full pass over frozen weights produces a distribution over the vocabulary.

TECHNICAL BREAKDOWNModule 5: LLM Interaction, Sampling & Constrained Generation

Inside the token loop: forward pass to appended token

Every generated token traverses the same pipeline: a full forward pass produces logits over the vocabulary; temperature rescales them; top-k and top-p truncate the candidate set; one sample is drawn; the token appends to the context. The loop is serial, and every stage between logits and sample is negligible next to the pass itself.

Logits Are Raw Scores

The final projection emits one score per vocabulary entry (~128k floats). Everything after is cheap vector arithmetic on a few dozen candidates.

Filter Order Matters

Temperature reshapes before truncation: at t→0 the distribution sharpens toward argmax regardless of top-k. Filters compose — penalties stack outside them.

The Append Is the State Change

Nothing else about the model changes between steps. Growth happens in the KV cache; that is the loop's only persistent mutation.

MATHEMATICAL FORMULATION · THE CHAIN
p_i = softmax(z_i / t) → keep top-k / nucleus-p → renormalize → sample → append

At t=1 the chain is the model's honest distribution; at t=0.1 it concentrates ~90% of mass on the top token; top-p=0.9 adapts the candidate count to distribution shape instead of fixing it.

REAL-WORLD PRODUCTION ENGINEERING
  • Sampling kernels are memory-trivial but latency-visible at batch 1: fused sampling kernels avoid a separate sync per step.
  • Engines expose per-request seeds for reproducibility — the sampler is deterministic given the seed and the logits.