TRAINING AT SCALE

MODULE 5: PRECISION & ACCELERATION · SCENE 19

Forget on purpose, recompute later

Activation checkpointing trades a third of compute for gigabytes of activation memory.

ACTIVATION VRAM: 13.7 GB

Activation Checkpointing drops intermediate layer activations and recomputes them during backprop, trading +33% compute to save gigabytes of VRAM.

ARCHITECTURE & FORMULATIONModule 5: Numerical Stability, Precision, & Acceleration

Activation Checkpointing & FlashAttention Mechanics

Storing all layer activations during the forward pass consumes tens of GBs of VRAM. Activation Checkpointing drops intermediate layer activations and recomputes them on-the-fly during the backward pass.

Trading Compute for Memory

By storing activations only at segment boundaries (e.g. every sqrt(L) layers), activation memory drops from O(L) to O(sqrt(L)). The cost is ~33% additional forward pass compute time.

FlashAttention-2 / 3 (Tiled Compute)

Standard attention writes N x N score matrices to slow GPU HBM. FlashAttention tiles Query, Key, and Value blocks inside fast GPU SRAM (20 TB/s bandwidth), computing online Softmax without ever materializing the quadratic matrix in HBM.

Selective Activation Recompute

Recomputing only memory-heavy, compute-cheap ops (like RMSNorm, SwiGLU, or Attention Dropout) saves ~70% memory with less than 10% compute penalty.

MATHEMATICAL FORMULATION · FLASHATTENTION ONLINE SOFTMAX REDUCTION
m_new = max( m_old, m_block ) d_new = d_old · exp( m_old - m_new ) + d_block · exp( m_block - m_new )

Online Softmax tracks running row maximums and rescale factors across SRAM tiles without full matrix materialization.

REAL-WORLD PRODUCTION ENGINEERING
  • FlashAttention-3 achieves over 800 TFLOPS (75%+ MFU) on H100 GPUs by leveraging FP8 Tensor Cores and asynchronous CUDA WGMMA instructions.
← 18 · Sixteen bits, carefully placed
20 · AdamW and the cosine slide →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]