WORLD 03 TRAINING AT SCALE
Watch information become a change in the model.
Pre-training across 6 comprehensive modules: from self-supervised objectives and web-scale data curation, to 3D parallelism, precision numerics, and run diagnostics.
JOURNEY · 6 MODULES · 21 SCENES
From random noise to a trained foundation model.
Objectives & Paradigm
What a model is actually asked to do, and why next-token prediction won.
01 · OBJECTIVES
A model is grown, not built
Weights start random. Every step shows the model one batch and nudges it — billions of times.
OPEN EXHIBIT →02 · OBJECTIVES
Two ways to hide the answer
Causal LM predicts every next token; masked LM fills blanks. The objective decides everything downstream.
OPEN EXHIBIT →03 · THE LOOP
Forward, backward, update
One training step in three beats: predict, attribute blame, move the weights.
OPEN EXHIBIT →04 · ATTENTION
The mask that lets time flow one way
Causal masking zeroes half the attention map — the mechanism behind autoregression, and a free compute win.
OPEN EXHIBIT →05 · CONTEXT
The square the model must pay for
Attention cost and memory grow with N². The context window is an arithmetic promise, not a marketing number.
OPEN EXHIBIT →Data at Web Scale
The pipeline that turns a multi-terabyte crawl into training tokens.
06 · DATA
From crawl to corpus
Heuristic filters — symbol ratios, language ID, text density — kill most of the web before a model sees it.
OPEN EXHIBIT →07 · DATA
Delete first, train later
MinHash + LSH find near-duplicates so the model memorizes the world once, not a million copies of it.
OPEN EXHIBIT →08 · TOKENIZER
Building the vocabulary
Byte-pair encoding grows a vocabulary from raw bytes — merge by merge, with rules that shape every token.
OPEN EXHIBIT →The Pre-Train Recipe
Scaling laws, RoPE, normalization, attention variants — the architectural spec.
09 · BUDGETING
Spend tokens like money
Chinchilla says the compute-optimal ratio is ~20 tokens per parameter. Plan the run before burning it.
OPEN EXHIBIT →10 · POSITION
Position as rotation
RoPE encodes position by rotating query and key pairs — relative distance falls out of the dot product.
OPEN EXHIBIT →11 · STABILITY
RMSNorm and SwiGLU
The small layers that keep a 100-layer network trainable: normalize, then gate.
OPEN EXHIBIT →12 · ATTENTION
Sharing the KV cache
Grouped-query attention lets query heads share K/V heads — cheaper memory with almost no quality loss.
OPEN EXHIBIT →Distributed Infrastructure
Hardware fabric, then the 3D parallelism that fills it.
13 · MEMORY
The state you pay to keep
Weights are the cheap part. Gradients and Adam moments quadruple the bill before a single token flows.
OPEN EXHIBIT →14 · PARALLELISM I
Copy everything, average the blame
Data parallelism clones the model per GPU and all-reduces gradients every step.
OPEN EXHIBIT →15 · COLLECTIVES
The ring that averages the world
N GPUs trade gradient chunks around a ring — each sends and receives exactly twice.
OPEN EXHIBIT →16 · PARALLELISM II
Slice the matrix itself
Tensor parallelism splits every matmul across GPUs and syncs twice per layer.
OPEN EXHIBIT →17 · PARALLELISM III
An assembly line of layers
Pipeline parallelism streams micro-batches through stage slices — and pays a bubble tax.
OPEN EXHIBIT →Precision & Acceleration
Numerics, memory-bound kernels, and the recompute trade.
Execution & Diagnostics
Optimizers, schedules, initialization, telemetry, and surviving a crash.
20 · OPTIMIZATION
AdamW and the cosine slide
Decoupled weight decay, bias-corrected moments, warmup, and a learning rate that coasts downhill.
OPEN EXHIBIT →21 · TELEMETRY
Watching a run breathe
Gradient norms, throughput, loss spikes, and the checkpoint machinery that survives a dead node.
OPEN EXHIBIT →