AI ENGVisual Encyclopedia

MODULE 3: THE PRE-TRAIN RECIPE · SCENE 09

Spend tokens like money

Chinchilla says the compute-optimal ratio is ~20 tokens per parameter. Plan the run before burning it.

Chinchilla compute-optimal training scales parameters N and tokens D equally (D ≈ 20N).

TECHNICAL BREAKDOWNModule 3: Modern Architectural Specifications (The Pre-Train Recipe)

Chinchilla Scaling Laws & Compute-Optimal Budgeting

DeepMind's Chinchilla study (Hoffmann et al.) corrected Kaplan's earlier scaling laws, proving that for compute-optimal training, model parameters N and training tokens D should scale in equal proportion: N ∝ D.

Chinchilla Multiplier (D ≈ 20N)

To train a compute-optimal model, allocate ~20 tokens per model parameter. A 7B model requires 140B tokens; a 70B model requires 1.4T tokens for compute efficiency.

Inference-Optimal Over-Training

Modern open models deliberately break Chinchilla optimality by over-training smaller models (e.g., Llama 3 8B trained on 15T tokens = 1,875 tokens/param). Paying higher pre-training cost yields cheaper inference forever.

Compute Budget Formula (6ND)

Total training FLOPs for a dense transformer is approximately 6 · N · D (2ND forward pass + 4ND backward pass).

MATHEMATICAL FORMULATION · POWER LAW LOSS FUNCTION
Loss(N, D) = E + ( A / N^a ) + ( B / D^b )

Cross-entropy loss L scales as power laws of parameter count N and dataset size D, approaching irreducible entropy E.

REAL-WORLD PRODUCTION ENGINEERING
  • Llama 3 8B trained on 15T tokens achieved benchmark performance competing with 70B Chinchilla-optimal models.