AI ENGVisual Encyclopedia

SCENE 23 / 24 · PRODUCTION OPERATIONS, DEPLOYMENT & FINOPS

The cold start tax

Container pull plus weight loading decides whether your first request survives.

FIRST TOKEN DELAY · 66.0 s
22.0sIMAGE PULL38.0sWEIGHT LOAD6.0sKV / WARMUPscale-to-zero economics vs p95 latency promises

Multi-GB weight loads dwarf container starts. Snapshots, tiered model caches and warm pools are how serverless inference survives its own first request.

TECHNICAL BREAKDOWNModule 8: Production Operations, Deployment & FinOps

The serverless cold start: container pull to parameter load

Serverless AI endpoints scale to zero when idle to save costs. However, waking up a cold container requires pulling multi-gigabyte container images, loading tens of gigabytes of model weights into GPU memory, and running CUDA warmup iterations — creating massive cold-start delays (10s to 60s+).

The Cold Start Chain

Image Pull (Docker layers) + Container Init + PyTorch/CUDA Runtime Init + Model Weight Load (HBM) + KV Cache Pre-allocation + Warmup Pass.

Fast Weight Loading

Using safetensors and mmap (memory-mapped files) allows direct DMA transfers from NVMe SSD to GPU VRAM, bypassing CPU serialization.

Warm Pools & Scale-to-1

Mitigating cold starts requires keeping warm standby instances or keeping 1 replica active, balancing cost vs p95 response time.

MATHEMATICAL FORMULATION · COLD START BREAK-DOWN
t_cold = t_container_pull + (Model_size_GB / NVMe_BW_GBs) + t_cuda_warmup

Loading a 14 GB model from NVMe at 3.5 GB/s takes 4s for weight transfer alone, on top of 15s container pull and 5s CUDA context setup = 24s total cold start.

REAL-WORLD PRODUCTION ENGINEERING
  • KServe, Modal, and Replicate use streaming container filesystems (e.g., CDFS) and pre-warmed GPU pools to reduce cold starts under 2 seconds.
  • Safetensors format is standard across Hugging Face because it enables zero-copy mmap loading directly to GPU.
← 22 · KV caches across nodes
24 · Every slice of the GPU bills →