Kimi K3 · mechanism 01 · executable lesson

A memory that can forget, correct, and retrieve

Kimi Delta Attention replaces a growing list of past keys and values with a fixed matrix. Follow one real PyTorch trace as each token transforms that memory.

The problemSoftmax attention keeps a KV entry for every past token. KDA asks whether a fixed state can retain the useful associations—and revise them when new evidence arrives.
St−1→S̄ = Diag(α)S→p = S̄ᵀk→e = v − p→ΔS = βkeᵀ→St = S̄ + ΔS→y = Stᵀq→handoff

×

+

changing valuecomputed resultactive vector component
What K3 changesSequence-length-growing KV storage becomes one fixed dk × dv state per head.
Why the delta rule mattersThe write stores the error, not the full value, so an aligned key corrects its previous prediction.
What the miniature provesUninterrupted and 3-token state-carrying execution match exactly: max error 0.