INFERENCE ENGINEERING

SCENE 11 / 24 · THE RUNTIME & MEMORY MANAGEMENT LAYER

End the fragmentation tax

Reserve exact blocks on demand instead of worst-case contiguous slabs.

TOKENS STORED · 0/64INTERNAL FRAG · 64 SLOTS
BLOCK TABLE MAPS LOGICAL POSITIONS → PHYSICAL PAGES · FRAGMENTATION ≤ 7 SLOTS PER SEQUENCE

Contiguous reservation bets every request hits max context. Paging allocates 8-token blocks on demand — waste drops from whole slabs to less than one block per sequence.

ARCHITECTURE & FORMULATIONModule 4: The AI Runtime & Memory Management Layer

PagedAttention: virtual memory for attention

Pre-allocation reserves worst-case contiguous slabs per request; most sit empty, and fragmentation strands the rest. PagedAttention borrows the OS answer: fixed-size blocks allocated on demand, a block table mapping logical positions to scattered physical pages, and copy-on-write for shared prefixes.

Fragmentation Kill

Reserved-but-unused KV (internal fragmentation) plus unallocatable gaps (external) historically wasted 60-80% of KV memory. Paging cuts waste to under 4%.

Block Tables

Each request carries a table: logical block i → physical block. Attention kernels gather K/V through indirection instead of assuming contiguity.

Copy-on-Write Forks

Two requests sharing a prefix share physical blocks; divergence triggers copy. Beam search and n-best sampling become nearly free in memory terms.

MATHEMATICAL FORMULATION · WASTE ACCOUNTING
waste = internal_frag (≤ block_size − 1 tokens) + external_frag (unusable gaps)

With 16-token blocks, internal waste is <15 tokens per request regardless of context length. vLLM's paper reports 2-4× throughput over naive allocation purely from this.

REAL-WORLD PRODUCTION ENGINEERING
  • Block size is a tuning knob: 16 tokens (vLLM default) balances kernel overhead against fragmentation.
  • PagedAttention made beam search economically viable again — pre-paging, it was a memory multiplier.
← 10 · Half a megabyte per token
12 · Reuse what shared prompts pay for →
Agent-native web · research mode

Experimental WebMCP · evolving community-group draft, not a W3C Standard or on the Standards Track.

Native status: Initializing. Registered: none

DOM → inferred capabilities · WebMCP → declared capabilities. Tools read real CUDA/Inference Atlas content. Calls may highlight temporary UI state. No backend or remote telemetry.

Local harness verifies application logic; it does not demonstrate native discovery or a browser agent.

Input schema
{
  "type": "object",
  "properties": {
    "query": {
      "type": "string",
      "minLength": 1,
      "maxLength": 300
    },
    "limit": {
      "type": "integer",
      "minimum": 1,
      "maximum": 10
    }
  },
  "required": [
    "query"
  ],
  "additionalProperties": false
}
Local log: last 100 calls while this panel is open. Inputs may contain personal text; review before sharing.
 
[]