HCA: Heavily Compressed Attention (toy implementation)¶
Paper Section 2.3.2 · Code: HCA in src/deepseek_v4_lab/attention/csa.py
Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
Why this exists
Problem: the same memory explosion as CSA, but some layers can tolerate a much cheaper context representation if coarse gist survives.
Response: pool m′=128 tokens per entry and run dense attention over compressed entries: no indexer, no top-k, maximal cheapness.
New cost: extreme blending destroys rare-token signal; past a pooling threshold the entry becomes undecodable.
Our experiment: Exp03 sweeps m′ and finds a sharp cliff between m′=8 and m′=16 on max-entropy data [MEASURED].
Exchange rate: COMPRESSION buys memory and bandwidth; spends fine-grained information.
What is implemented [IMPLEMENTED]¶
- Single-series pooling of m′ tokens per entry (Eqs 20–23), no overlap, content-dependent softmax weights + learnable positional bias.
- Dense attention over all causally visible compressed entries (no indexer, no top-k) + sliding-window branch, MQA shared KV, RMSNorm, partial RoPE, attention sink, grouped output projection, same treatment as CSA.
Toy-scale result. Direction is informative; production magnitude is not.
Measured results (Exp03, needle task)¶
| m′ | accuracy |
|---|---|
| 8 | 0.352 |
| 16 | 0.055 (chance) |
| 32 | 0.068 |
| 64 | 0.068 |
Sharp cliff between m′=8 and m′=16 at toy training budget: once entries average ≥16 max-entropy tokens, the rare-token signal cannot be recovered from the normalized blend [MEASURED].
Interpretation [INFERRED]¶
This cliff is precisely why V4 does not rely on HCA alone: heavy compression destroys fine-grained retrieval but retains coarse "gist" cheaply. The hybrid architecture lets HCA layers carry bulk context while CSA layers carry precise access paths.
Limitations¶
- Longer training / larger c would soften the cliff; we did not tune for it.
- Real text compresses far better than i.i.d. noise (redundancy structure); m′=128 in production vs our chance-level toy numbers are not comparable.