Skip to content

mHC: Manifold-Constrained Hyper-Connections (Exp07)

Paper Section 2.2 · Code: src/deepseek_v4_lab/residual/mhc.py

Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger

Why this exists

Problem: widening the residual stream into multiple learned lanes lets deep stacks amplify signal exponentially; unconstrained mixing explodes with depth.

Response: constrain every lane-mixing matrix to the Birkhoff polytope (doubly stochastic) via differentiable Sinkhorn-Knopp projection.

New cost: representational freedom of the mixing is capped by σ(B) ≤ 1; extra parameters and projection work per layer.

Our experiment: Exp07 verifies non-expansiveness, closure under multiplication, and stability to depth 128 [IMPLEMENTED + MEASURED].

Exchange rate: CONSTRAINT buys depth-stability guarantees; spends mixing flexibility.

Implemented [IMPLEMENTED]

  • Standard residual, unconstrained HC, and Birkhoff-constrained mHC behind one interface; dynamic parameter generation per Eqs 3–5 (RMSNorm(vec(X)) → linear + static bias + small-init gate), A=σ(·), C=2σ(·) per Eqs 6–7.
  • Sinkhorn-Knopp projection (exp → alternating row/col normalize, t=20) differentiable end-to-end [Eq 8].

Verified properties (unit tests)

  • Output doubly stochastic, non-negative [PAPER Eq 2]
  • σ_max(B) ≤ 1 ⇒ non-expansive residual mapping
  • Closure under multiplication, the deep-stack guarantee
  • Plain mode reduces exactly to x + F(norm(x))

Measured (Exp07)

quantity HC mHC
single-layer σ_max (adversarial init) 5.07 1.00

Forward/backward norms tracked to depth 128 in artifacts. With default small init both modes are numerically stable at toy depth; the constraint's value is the guarantee (‖∏B_l‖₂ ≤ 1 by closure), not an observable difference under gentle initialization [INFERRED, consistent with PAPER Section 2.2 motivation].

Limitations

  • Stability study uses MLP inner functions, not attention layers.
  • The identity-bias added before Sinkhorn (+4·I) is our initialization choice for lane preservation; paper does not specify init.