DeepMind's Chinchilla study (Hoffmann et al.) corrected Kaplan's earlier scaling laws, proving that for compute-optimal training, model parameters N and training tokens D should scale in equal proportion: N ∝ D.
Chinchilla Multiplier (D ≈ 20N)
To train a compute-optimal model, allocate ~20 tokens per model parameter. A 7B model requires 140B tokens; a 70B model requires 1.4T tokens for compute efficiency.
Inference-Optimal Over-Training
Modern open models deliberately break Chinchilla optimality by over-training smaller models (e.g., Llama 3 8B trained on 15T tokens = 1,875 tokens/param). Paying higher pre-training cost yields cheaper inference forever.
Compute Budget Formula (6ND)
Total training FLOPs for a dense transformer is approximately 6 · N · D (2ND forward pass + 4ND backward pass).
MATHEMATICAL FORMULATION · POWER LAW LOSS FUNCTION
Loss(N, D) = E + ( A / N^a ) + ( B / D^b )
Cross-entropy loss L scales as power laws of parameter count N and dataset size D, approaching irreducible entropy E.
REAL-WORLD PRODUCTION ENGINEERING
Llama 3 8B trained on 15T tokens achieved benchmark performance competing with 70B Chinchilla-optimal models.