Million-Token Context: economics summary (Exp05)¶
Question [spec Section 6]: what does 1M-token context cost under each memory architecture? Never "our model supports 1M tokens."
Evidence tags: PAPER stated by DeepSeek · DERIVED mathematically implied · IMPLEMENTED reproduced in this project · MEASURED observed in our experiments · INFERRED our interpretation · UNKNOWN not established. Full definitions: evidence ledger
KV cache per layer at 1M tokens (FP16, toy dims c=32)¶
| architecture | MB | vs vanilla |
|---|---|---|
| vanilla GQA-like | 1073.7 | 1× |
| CSA m=4 | 50.3 | 21× |
| hybrid CSA+HCA | 25.7 | 42× |
| HCA m′=128 | 1.1 | ~1000× |
[DERIVED from analytic formulas; slopes validated by measured RSS growth up to 192K tokens on a 3.7GB-RAM machine, labeled extrapolation beyond]
Reading¶
- Compression does not eliminate the memory problem; it converts it into a retrieval-quality problem. The ladder makes the exchange rate visible.
- The paper's production numbers (V4-Pro = 10% of V3.2 KV at 1M) live on the same frontier we measure at toy scale [PAPER Section 1 vs MEASURED Exp05].
Method notes¶
- Measured allocations chunked at 16K tokens to respect RAM ceiling.
- FLOPs curves logged alongside (vanilla quadratic term dominates from ~16K tokens onward at these dims).