SCENE 03 / 24 · FOUNDATIONS OF INFERENCE & EXECUTION
TTFT, TPOT and goodput
The three numbers every inference SLO is made of.
TOKENS OUT · 0
Users feel TTFT as responsiveness and TPOT as reading speed. Goodput only counts responses that satisfy BOTH budgets under real load.
ARCHITECTURE & FORMULATIONModule 1: Foundations of Inference & Execution
TTFT, TPOT, and the goodput discipline
Users feel two distinct clocks: how fast the first word appears (TTFT) and how fast the rest streams (TPOT/ITL). End-to-end latency is their sum over the output length. Throughput counts tokens; goodput counts only responses that met both budgets — the number that actually maps to revenue and SLA credits.
The SLO Split
Product teams promise TTFT < 500ms and TPOT < 50ms far more often than a single E2E number. The split matters because the phases have different bottlenecks and different fixes.
Goodput vs Throughput
A batch of 256 that pushes TPOT to 300ms produces huge throughput and near-zero goodput. Maximizing goodput means sizing batches to the SLO envelope, not to GPU saturation.
Budget Accounting
E2EL = network + queue + tokenize + prefill + N × decode + detokenize. Every middleware hop spends latency the model budget doesn't have.
Throughput can rise monotonically with batch size while goodput collapses. The optimal operating point is the largest batch inside the SLO envelope — which is a scheduling problem, not a hardware one.
REAL-WORLD PRODUCTION ENGINEERING
Frontier API providers publish TTFT and TPOT separately; internal dashboards should too, per model and per tenant.
Latency percentiles (p50/p95/p99) beat means: tail TTFT under load is where user-perceived slowness hides.