Causal Language Modeling (CLM) vs. Masked Language Modeling (MLM)
The choice of self-supervised training objective fundamentally dictates model architecture, inference latency, and generative capabilities. Masked Language Modeling (BERT) bidirectionally inspects context to fill in hidden tokens, while Causal Language Modeling (GPT / Llama) strictly predicts the next token given preceding context.
CLM Mechanics (Autoregressive)
CLM computes cross-entropy loss on every single token position in a sequence simultaneously using a lower-triangular causal attention mask. The model learns P(x_t | x_1, x_2, ..., x_{t-1}).
MLM Mechanics (Bidirectional)
MLM replaces ~15% of tokens with a [MASK] token. The model sees past and future tokens for unmasked tokens, computing loss only over the masked locations. While powerful for encoder tasks, it cannot generate long text autoregressively without N passes.
Why CLM Won the Generative Era
CLM computes N predictions per forward pass (one for each position in the context), making token-for-token training computational efficiency far higher than MLM (which only calculates loss on the 15% masked positions).
The global loss is the mean negative log-likelihood of predicting each true token x_i given all prior tokens in the sequence.
- Every modern foundation model (GPT-4, Claude 3.5, Llama 3, Qwen 2.5) uses CLM pre-training.
- Hybrid models like T5 or UL2 attempt unified objectives, but pure CLM remains the undisputed standard due to KV-cache inference compatibility.