Hybrid Attention: why two compressed regimes¶
Paper Section 2.3, Section 2.3.4 · Code: TinyRetrievalLM(kind="hybrid"), Exps 01–04
Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
Why this exists
Problem: no single compression ratio serves every information-access pattern; precise retrieval and bulk gist want opposite designs.
Response: interleave layers: cheap heavily-pooled HCA layers carry bulk context while precise CSA layers carry exact access paths.
New cost: two mechanisms to train and schedule; inside the full stack the indexer's job gets harder.
Our experiment: Exp04's four-way frontier table shows hybrid matching most of CSA's accuracy at half its KV cost [MEASURED].
Exchange rate: COMPRESSION buys memory per layer; spends per-layer retrieval precision.
The question [spec Section 5]¶
Why are two compressed attention regimes better than forcing one mechanism to solve every information-access problem?
Toy-scale result. Direction is informative; production magnitude is not.
Four-way comparison (Exp04, same budget, needle task)¶
| variant | accuracy | KV @1M ctx (FP16 est.) |
|---|---|---|
| vanilla GQA-like | 0.879 | 1024 MB |
| CSA (m=4, k=8) | 0.238 | 48 MB |
| HCA (m′=32) | 0.062 | 4 MB |
| hybrid (interleaved) | 0.207 | 26 MB |
[MEASURED accuracies; KV numbers DERIVED from analytic formulas validated in tests]
Reading¶
- The memory axis behaves exactly as the paper claims: 21× reduction for CSA-style layers, 256× for HCA-style, hybrid in between.
- Hybrid matches most of CSA's retrieval accuracy at roughly half its KV cost: alternating cheap HCA layers with precise CSA layers dominates either pure regime on the accuracy-per-byte frontier at toy scale.
- The remaining gap to vanilla is the honest cost of compression under an adversarial (max-entropy) synthetic distribution [INFERRED].
Baseline cost curve (Exp01)¶
Quadratic FLOPs + linear KV growth confirmed analytically to 1M tokens and empirically to 4K on CPU; this is the curve every compressed variant bends.
Limitations¶
- Single tiny task, 2-layer models, one seed per cell (seeded, but no error bars). Directional conclusions only.
- FLOPs comparisons are analytic estimates, not measured wall-clock.