FinOps, MIG slicing, spot pools and autoscaling economics
Production AI serving cost is dominated by idle GPU capacity. Multi-Instance GPU (MIG) hardware slicing partitions a single H100 into up to 7 isolated GPU instances for smaller models. Combining MIG with spot instance pools, dynamic autoscaling on queue depth, and batch tuning minimizes cost per million tokens.
MIG Hardware Partitioning
NVIDIA MIG divides GPU compute cores, crossbar paths, and memory bandwidth into hardware-isolated instances with guaranteed QoS.
Spot Instance Economics
Cloud spot/preemptible GPUs offer 60-80% discounts. Serving architectures handle spot terminations by draining KV caches and re-routing active requests.
Autoscaling Metrics
Autoscaling on CPU/GPU utilization fails because GPUs report 100% load during memory waits. Autoscale on queue depth and ITL instead.
An H100 costing $3.50/hr spot generating 2,400 tok/sec achieves $0.40 per million tokens. On-demand ($10/hr) costs $1.15 per million tokens for the exact same output.
- Karpenter and KEDA scale Kubernetes GPU worker pools based on pending request queues and p95 TTFT metrics.
- Anyscale, Baseten, and Together AI run multi-tenant MIG and spot clusters to optimize infrastructure margin.