SCENE 04 / 24 · HARDWARE MICRO-ARCHITECTURE & SILICON
CPU, GPU, TPU, LPU
Every substrate makes a different bet on parallelism, memory and flexibility.
Compute and bandwidth rarely peak together. The LPU's 80 TB/s SRAM fabric exists precisely to keep a deterministic pipeline fed; the GPU trades some of that for flexibility.
CPUs, GPUs, TPUs, LPUs and inference ASICs all execute the same matmuls with radically different economics. Each substrate is a bet: flexibility (CPU), generality (GPU), dense systolic throughput (TPU), deterministic SRAM-resident scheduling (LPU), or cost-per-token within one vendor's cloud (Inferentia).
The Flexibility Tax
General-purpose silicon pays for branchy, irregular workloads with control overhead. Specialized silicon recovers that overhead as latency — until the model changes.
Memory Placement Is the Real Spec
The LPU's headline number is not FLOPs, it is 230 GB of on-die SRAM: weights stay resident and the memory wall disappears. HBM parts rent bandwidth instead.
Software Gravity
The GPU wins in production less on peak specs than on ecosystem: kernels, compilers, and a decade of framework integration. An ASIC without a compiler story is a benchmark trophy.
Decode time is weight streaming time. Substrates compete on bytes/s first and FLOPs second, which is why the LPU's 80 TB/s fabric beats the H100's 989 TF for small-batch decode.
REAL-WORLD PRODUCTION ENGINEERING
Groq's LPU schedules the entire model deterministically at compile time — no warp schedulers, no caches, no variance.
AWS Inferentia competes on $/token inside EC2, not on flexibility; model portability is the price.