INFERENCE ENGINEERING

SCENE 15 / 24 · INTERACTION, SAMPLING & CONSTRAINED GENERATION

Force valid JSON

Grammar constraints mask impossible tokens before the softmax ever runs.

GRAMMAR STATE · STARTexpect an object braceEMITTED · {
✗tok 0✗tok 1✗tok 2✗tok 3✗tok 4✗tok 5✗tok 6✗tok 7

Invalid tokens get -inf logits before the softmax — the model literally cannot emit a syntax error. Parse state advances per token; masking cost is negligible next to attention.

ARCHITECTURE & FORMULATIONModule 5: LLM Interaction, Sampling & Constrained Generation

Constrained decoding: grammars that make invalid tokens unselectable

Structured output forces the model to emit only tokens consistent with a grammar state. At each step, tokens that would break the parse get -inf logits before the softmax — the model cannot produce a syntax error because the syntax-violating tokens have zero probability.

Logit Masking

A per-state mask (allowed token set) multiplies logits before sampling. Masking is a vectorized op: microsecond-scale, invisible next to attention.

Grammar Compilation

Context-free grammars or JSON schemas compile to token-level state machines ahead of time. xgrammar and outlines precompile to bitmask lookups keyed by parse state.

The Semantics Caveat

Syntactic validity is guaranteed; semantic validity is not. A grammar can force valid JSON while the values inside are still wrong — validation belongs above the engine.

MATHEMATICAL FORMULATION · MASKED RENORMALIZATION
p_i = softmax(z)_i / Σ_{j∈allowed} softmax(z)_j for i ∈ allowed · p_i = 0 otherwise

Renormalization preserves the model's preferences among legal tokens. The masking engine's only real cost is maintaining parse state — O(1) per token with a precompiled automaton.

REAL-WORLD PRODUCTION ENGINEERING
  • SGLang uses xgrammar to cut guided-decoding overhead ~10× vs naive recompilation per request.
  • OpenAI-compatible 'response_format: json_schema' is this machinery surfaced as an API parameter.
← 14 · Shape the distribution
16 · Never leave a slot empty →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]