Causal Language Modeling (CLM) vs. Masked Language Modeling (MLM)
The choice of self-supervised training objective fundamentally dictates model architecture, inference latency, and generative capabilities. Masked Language Modeling (BERT) bidirectionally inspects context to fill in hidden tokens, while Causal Language Modeling (GPT / Llama) strictly predicts the next token given preceding context.
CLM Mechanics (Autoregressive)
CLM computes cross-entropy loss on every single token position in a sequence simultaneously using a lower-triangular causal attention mask. The model learns P(x_t | x_1, x_2, ..., x_{t-1}).
MLM Mechanics (Bidirectional)
MLM replaces ~15% of tokens with a [MASK] token. The model sees past and future tokens for unmasked tokens, computing loss only over the masked locations. While powerful for encoder tasks, it cannot generate long text autoregressively without N passes.
Why CLM Won the Generative Era
CLM computes N predictions per forward pass (one for each position in the context), making token-for-token training computational efficiency far higher than MLM (which only calculates loss on the 15% masked positions).
MATHEMATICAL FORMULATION · CAUSAL CROSS-ENTROPY LOSS