Chinchilla Scaling Laws & Compute-Optimal Budgeting
DeepMind's Chinchilla study (Hoffmann et al.) corrected Kaplan's earlier scaling laws, proving that for compute-optimal training, model parameters N and training tokens D should scale in equal proportion: N ∝ D.
Chinchilla Multiplier (D ≈ 20N)
To train a compute-optimal model, allocate ~20 tokens per model parameter. A 7B model requires 140B tokens; a 70B model requires 1.4T tokens for compute efficiency.
Inference-Optimal Over-Training
Modern open models deliberately break Chinchilla optimality by over-training smaller models (e.g., Llama 3 8B trained on 15T tokens = 1,875 tokens/param). Paying higher pre-training cost yields cheaper inference forever.
Compute Budget Formula (6ND)
Total training FLOPs for a dense transformer is approximately 6 · N · D (2ND forward pass + 4ND backward pass).
Cross-entropy loss L scales as power laws of parameter count N and dataset size D, approaching irreducible entropy E.
- Llama 3 8B trained on 15T tokens achieved benchmark performance competing with 70B Chinchilla-optimal models.