A model is grown, not built — Initialization to Signal
Neural network pre-training is a continuous optimization procedure where a multi-billion parameter weight tensor begins as random Gaussian noise and is iteratively shifted along loss gradients across trillions of tokens.
Initialization Variance
Weights are initialized using scaled normal distributions (e.g. Xavier/Glorot or Kaiming initialization). If initialization scale is too high, activations explode into NaN; if too low, gradients vanish before reaching deep early layers.
Accumulated Nudges
Each mini-batch calculates a loss score and computes partial derivatives ∂L/∂w for every parameter w. The optimizer applies a minuscule update step. Over 10^12 tokens, these individual nudges aggregate into structured semantic representations.
Inference vs. Training
During training, model weights are live target buffers requiring gradient history, momentum, and variance state. During inference, weights are frozen, requiring 4x to 8x less VRAM.
Each weight w is adjusted by learning rate η times the bias-corrected first moment estimate divided by the square root of the bias-corrected second moment estimate.
- Llama 3 70B was trained on over 15 trillion tokens using ~24,000 GPU clusters over several months.
- Early training stability is so fragile that the first 2,000-5,000 steps use a linear 'warmup' phase ramping learning rate from 0 to peak.