AI ENGVisual Encyclopedia

SCENE 15 / 24 · INTERACTION, SAMPLING & CONSTRAINED GENERATION

Force valid JSON

Grammar constraints mask impossible tokens before the softmax ever runs.

GRAMMAR STATE · STARTexpect an object braceEMITTED · {
✗tok 0✗tok 1✗tok 2✗tok 3✗tok 4✗tok 5✗tok 6✗tok 7

Invalid tokens get -inf logits before the softmax — the model literally cannot emit a syntax error. Parse state advances per token; masking cost is negligible next to attention.

TECHNICAL BREAKDOWNModule 5: LLM Interaction, Sampling & Constrained Generation

Constrained decoding: grammars that make invalid tokens unselectable

Structured output forces the model to emit only tokens consistent with a grammar state. At each step, tokens that would break the parse get -inf logits before the softmax — the model cannot produce a syntax error because the syntax-violating tokens have zero probability.

Logit Masking

A per-state mask (allowed token set) multiplies logits before sampling. Masking is a vectorized op: microsecond-scale, invisible next to attention.

Grammar Compilation

Context-free grammars or JSON schemas compile to token-level state machines ahead of time. xgrammar and outlines precompile to bitmask lookups keyed by parse state.

The Semantics Caveat

Syntactic validity is guaranteed; semantic validity is not. A grammar can force valid JSON while the values inside are still wrong — validation belongs above the engine.

MATHEMATICAL FORMULATION · MASKED RENORMALIZATION
p_i = softmax(z)_i / Σ_{j∈allowed} softmax(z)_j for i ∈ allowed · p_i = 0 otherwise

Renormalization preserves the model's preferences among legal tokens. The masking engine's only real cost is maintaining parse state — O(1) per token with a precompiled automaton.

REAL-WORLD PRODUCTION ENGINEERING
  • SGLang uses xgrammar to cut guided-decoding overhead ~10× vs naive recompilation per request.
  • OpenAI-compatible 'response_format: json_schema' is this machinery surfaced as an API parameter.
← 14 · Shape the distribution
16 · Never leave a slot empty →