POST-TRAINING & DISTILLATION

MODULE 8: ADVANCED CAPABILITIES & DEPLOYMENT · SCENE 23

Post-Training Quantization: GPTQ, AWQ, FP8

r-bit grouped weight error, quality retained, and the VRAM you actually need at bits-per-weight.

BITSGROUPQUALITY RETAINED · 98.9%
BF16 · 140 GB4-BIT · 35 GB (70B MODEL)err ≈ 2^−4 · √(1/128) → 98.9% RETAINED ✓

GPTQ compensates rounding error through the inverse Hessian; AWQ scales salient channels. Quantize the final aligned model, then re-eval.

ARCHITECTURE & FORMULATIONModule 8: Advanced Capabilities & Deployment

Post-training quantization: GPTQ, AWQ, FP8

Trained weights are too fat to serve. Post-training quantization compresses them after training, using calibration data to choose rounding that preserves the layer's function: GPTQ via iterative second-order error compensation, AWQ via activation-aware scaling, FP8 via calibrated ranges.

GPTQ

Quantize columns one at a time, redistributing each column's rounding error into the not-yet-quantized weights via the inverse Hessian. 4-bit with group size 128 is the sweet spot.

AWQ

Protect the ~1% of salient weight channels (large activations) by per-channel scaling instead of mixed precision — hardware-friendly, no outliers stored separately.

FP8 Calibration

Dynamic ranges need a calibration pass; too few calibration samples inflates KL divergence against the bf16 reference.

MATHEMATICAL FORMULATION · QUANTIZATION ERROR & RETAINED QUALITY
err ≈ 2^(−r) · √(1/group) · 0.5; retained ≈ 1 − 4·err; VRAM = params_B · bits/8

4-bit, group 128: err ≈ 0.0028 → ~98.9% quality retained. A 70B model: 140 GB at bf16 vs 35 GB at 4-bit — the difference between a node and a laptop.

REAL-WORLD PRODUCTION ENGINEERING
  • Quantize the final aligned model, not an earlier checkpoint — alignment is where behavior lives, and PTQ after alignment preserves it.
  • Always eval the quantized model on your task suite: per-layer error compounds non-uniformly, and 4-bit can cost 1–3 points on reasoning-heavy sets.
← 22 · Multimodal & Agentic Alignment
24 · Deployment: Distillation & Serving Prep →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]