TRAINING AT SCALE

MODULE 6: EXECUTION & DIAGNOSTICS · SCENE 21

Watching a run breathe

Gradient norms, throughput, loss spikes, and the checkpoint machinery that survives a dead node.

MFU %: 42.5%SPIKE DETECTED @ STEP 10 → REWIND ENGINE ACTIVE

Real-time telemetry detects loss spikes and gradient norm anomalies, triggering automated checkpoint rewinds.

ARCHITECTURE & FORMULATIONModule 6: Execution Dynamics & Diagnostics

Run Telemetry, Gradient Spikes, & Crash Recovery Engines

Pre-training across 20,000+ GPUs running continuously for 3 months suffers from inevitable hardware node crashes, cosmic ray bit flips, stragglers, and sudden loss spikes.

Monitoring Critical Telemetry Signals

Real-time dashboards track: 1. Loss curve & gradient norm ||g||. 2. Model FLOPs Utilization (MFU %). 3. Token throughput (tokens/sec/GPU). 4. HBM temperature & ECC memory errors.

Loss Spike Rewind Strategy

When loss spikes to infinity or NaN due to bad data shards or numerical explosion, automated recovery engines rewind to a checkpoint 2,000 steps prior, discard the bad data shard window, adjust LR, and resume.

Asynchronous Distributed Checkpointing

Saving a 70B model checkpoint (1.1 TB) every 1,000 steps uses background host RAM buffering and fast parallel NVMe pipelines to prevent blocking GPU computation.

MATHEMATICAL FORMULATION · MODEL FLOPS UTILIZATION (MFU)
MFU = ( 6 × N_params × Tokens_per_sec ) / ( Peak_TFLOPS_per_GPU × Total_GPUs )

Measuring the ratio of achieved theoretical matrix FLOPs against maximum hardware peak specs.

REAL-WORLD PRODUCTION ENGINEERING
  • Meta's Llama 3 405B training run sustained 38%+ MFU across 16,384 H100 GPUs while automatically recovering from over 400 node hardware failures without manual intervention.
← 20 · AdamW and the cosine slide
ATLAS →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]