INFERENCE ENGINEERING

SCENE 24 / 24 · PRODUCTION OPERATIONS, DEPLOYMENT & FINOPS

Every slice of the GPU bills

MIG slicing, spot pools and autoscaling turn utilization into run-cost.

SPOT MIX50% SPOT
MIG SLICES ON ONE H100$2.61 / M TOKENSfull on-demand: $3.79 → spot-adjusted: $2.61spot saves 31% — with retry-on-interruption overhead priced in

Fractional GPUs absorb small models cheaply; spot pools carry the checkpoint-and-drain tax; autoscale on concurrency, not CPU — GPUs lie about utilization.

ARCHITECTURE & FORMULATIONModule 8: Production Operations, Deployment & FinOps

FinOps, MIG slicing, spot pools and autoscaling economics

Production AI serving cost is dominated by idle GPU capacity. Multi-Instance GPU (MIG) hardware slicing partitions a single H100 into up to 7 isolated GPU instances for smaller models. Combining MIG with spot instance pools, dynamic autoscaling on queue depth, and batch tuning minimizes cost per million tokens.

MIG Hardware Partitioning

NVIDIA MIG divides GPU compute cores, crossbar paths, and memory bandwidth into hardware-isolated instances with guaranteed QoS.

Spot Instance Economics

Cloud spot/preemptible GPUs offer 60-80% discounts. Serving architectures handle spot terminations by draining KV caches and re-routing active requests.

Autoscaling Metrics

Autoscaling on CPU/GPU utilization fails because GPUs report 100% load during memory waits. Autoscale on queue depth and ITL instead.

MATHEMATICAL FORMULATION · COST PER MILLION TOKENS
Cost_per_1M = (Hourly_GPU_Rate / (Throughput_tok_sec × 3600)) × 1,000,000

An H100 costing $3.50/hr spot generating 2,400 tok/sec achieves $0.40 per million tokens. On-demand ($10/hr) costs $1.15 per million tokens for the exact same output.

REAL-WORLD PRODUCTION ENGINEERING
  • Karpenter and KEDA scale Kubernetes GPU worker pools based on pending request queues and p95 TTFT metrics.
  • Anyscale, Baseten, and Together AI run multi-tenant MIG and spot clusters to optimize infrastructure margin.
← 23 · The cold start tax
ATLAS →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]