Modifying token sampling probabilities in real time using grammar constraints, regex masks, and safety logit biases.
INTERACTIVE EXHIBIT 16
Constrained Decoding & Logit Masking
INFERENCE RUNTIME MASKING
STRICT FINITE-STATE AUTOMATON
Grammar transition tables dynamically mask out tokens that violate the required schema.
💡 Near-Zero Latency: Pre-indexed FSA tables in CUDA kernels execute during logit softmax with zero throughput penalty.
ARCHITECTURAL OVERVIEW
Constrained Decoding & Logit Filtering
Rather than hoping a model follows safety instructions in natural language, Constrained Decoding mathematically enforces safety rules directly during token sampling. By modifying logit distributions before softmax, the inference engine guarantees that disallowed tokens or invalid grammar states have zero probability of generation.
Logit Bias Suppression
Disallowed vocabulary tokens (e.g., toxic phrases, PII numbers, forbidden command strings) have their logits set to -infinity, making their generation mathematically impossible.
Grammar-Guided State Machines (CFGs / Regex)
Finite-state automata (FSAs) track the valid token transition matrix. At each decoding step t, tokens that violate the allowable Context-Free Grammar or regex pattern are dynamically masked out.
Zero-Overhead Sampling Kernels
Modern inference runtimes (e.g., Outlines, Guidance, vLLM) pre-index token transition tables, executing constrained sampling directly in CUDA kernels with zero throughput degradation.
MATHEMATICAL FORMULATION · CONSTRAINED SOFTMAX TOKEN PROBABILITY
Where z_v is the raw logit, s_t is the current grammar state, and m_v is the binary safety mask. Setting m_v = -infinity guarantees that token v has exactly zero probability of being sampled.
REAL-WORLD PRODUCTION ENGINEERING
Use constrained decoding for structured JSON outputs to guarantee valid schemas and prevent prompt injection tag breakouts.
Apply token-level PII masks to suppress the generation of 16-digit credit card patterns or Social Security numbers.
Combine constrained decoding with speculative decoding: verify drafted tokens against the grammar state machine in parallel.