The filters: temperature, top-k, top-p on real logits
Three knobs sculpt one distribution. Temperature divides logits — sharpening or flattening the whole shape. Top-k keeps a fixed count of candidates. Top-p keeps the smallest set whose mass exceeds a threshold, adapting to distribution shape: wide when uncertain, narrow when confident.
Temperature Is a Scale
z/t with t<1 sharpens, t>1 flattens. At t→0 it approaches argmax; entropy of the distribution falls monotonically as t falls.
Top-k vs Top-p
Fixed k wastes candidates on confident steps and under-covers on flat ones. Nucleus sampling's set size breathes with the distribution — the reason it became the default.
Hardware Cost
Sorting 128k logits for top-k is a real kernel (radix or bitonic); nucleus accumulation is a scan. Both are trivial next to the 14 GB weight stream — but not free at high QPS.
With BASE_LOGITS at top-p=0.5, three tokens survive; at top-p=0.95, six do. The kept set's mass is guaranteed ≥ p — that's the 'nucleus' contract.
- Ampere+ GPUs run sampling as fused single kernels; older stacks synced per filter, adding ~1 ms/step at batch 1.
- Repetition and presence penalties apply outside this math (on counts, not logits) but stack with it in the same pass.