Block Attention Residuals (AttnRes) Softmax Depth Routing

Replaces linear additions (h_l = h_{l-1} + f(h)) with softmax attention weighting across block representations b_0..b_3



Dynamic Depth Attention Distribution alpha_{i -> l}
Mathematical Formula Mechanic
alpha_{i -> l} = softmax( (w_l * RMSNorm(b_i)) / sqrt(d_model) )

h_l = sum_{i=0}^{N-1} alpha_{i -> l} * b_i

Unlike standard residual connections which blindly sum activations, AttnRes dynamically queries which prior layer representations contain relevant features for the current depth block.