INFERENCE ENGINEERING

SCENE 07 / 24 · TRANSFORMATION, COMPILATION & QUANTIZATION

The precision ladder

FP32 down to FP4: what each rung saves, and what calibration rescues.

7B MODEL · 7.0 GB (was 28.0)
● FP32 weight▬ dequantized— rounding errormax err · 0.0035 · scale 0.0072

PTQ in one picture: pick a scale from calibration data, snap each weight to the grid, and the gap between teal and gold is your quality bill. Lower rungs widen the grid — and the error.

ARCHITECTURE & FORMULATIONModule 3: Model Transformation, Compilation & Quantization

Precision reduction: the size-fidelity exchange rate

Quantization maps FP32 weights onto coarser grids: FP16/BF16 halve storage losslessly for most workloads; INT8 needs calibrated scales; INT4/FP4 make quality a real engineering conversation. Post-Training Quantization does all of this without retraining — the model is frozen, and scale factors absorb the compression.

PTQ Mechanics

Calibrate on a few hundred representative samples; per-channel scales = max|w| / grid_max. The rounding error budget per weight is half a grid step.

Weight-Only vs Weight-and-Activation

W4A16 keeps activations high precision and wins on quality; W8A8 unlocks integer Tensor Core paths and wins on speed. KV cache quantization is a third axis with its own trade.

Outlier Channels

A handful of activation channels carry magnitude 20-100× the median. Naive per-tensor scales waste the grid on them; per-channel or AWQ-style salience weighting rescues quality.

MATHEMATICAL FORMULATION · SYMMETRIC PTQ
scale = max|w| / (2^(b-1) − 1) · q = round(w / scale) · VRAM = params × b/8

A 7B model at FP32 is 28 GB; INT4 is 3.5 GB — an 8× reduction with max per-weight error ≤ scale/2. Whether that error compounds into gibberish or disappears is what calibration decides.

REAL-WORLD PRODUCTION ENGINEERING
  • GPTQ quantizes layer-by-layer against calibration activations, minimizing output error rather than weight error.
  • KV cache quantization (FP8) doubles usable context on the same silicon — often the highest-leverage rung for long-context serving.
← 06 · Warps and Tensor Cores
08 · Fuse the graph →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]