Modern architecture recipes replaced standard LayerNorm with RMSNorm to eliminate mean-centering overhead, and replaced ReLU/GeLU with SwiGLU gated activations for superior gradient propagation.
RMSNorm (Root Mean Square Norm)
RMSNorm normalizes activation inputs using root mean square without calculating mean offset. This saves ~7% to 10% kernel latency per transformer block with zero loss in training stability.
SwiGLU Activation Function
SwiGLU combines Swish activation with Gated Linear Units: SwiGLU(x) = (xW + b) * Swish(xV + c). It provides smoother gradients and higher expressive capacity.
Pre-Normalization Architecture
Placing RMSNorm BEFORE attention and MLP blocks (Pre-LN) instead of after (Post-LN) keeps the residual main branch uncorrupted, preventing vanishing gradients in 80+ layer deep networks.