Deployment: distillation and serving prep
The last mile converts a good model into a shippable one: distill capability into a smaller student, calibrate for FP8 serving, and verify the deployed artifact reproduces the evaluated behavior — not a nearby cousin.
KL Distillation
The student minimizes KL against the teacher's softened distribution (temperature T), inheriting the teacher's dark knowledge — relative probabilities — not just its argmax.
Calibration Sets
FP8 serving quantization uses a few hundred representative samples to fix activation ranges. Too few samples inflates output KL versus the reference.
Evals on the Shipped Artifact
Re-run the full eval suite on the quantized, exported model. The artifact you serve is the artifact you score.
The T² factor keeps gradient magnitudes comparable across temperatures. A 7B model at 4-bit needs ~3.5 GB of weights — plus 15–40% for KV cache and runtime overhead depending on batch and context.
- Sequence-level distillation (train the student on the teacher's sampled outputs) usually beats pure logit distillation for aligned behavior.
- Ship with the calibration recipe recorded: samples, quantization scheme, and the eval delta vs bf16 — reproducibility is part of the artifact.