Radix trees and prefix caching: compute the shared prompt once
System prompts, few-shot headers and multi-turn history are recomputed by thousands of requests that differ only in their tails. A radix tree of cached KV blocks matches the longest shared prefix automatically, and every hit converts prefill compute into an O(1) pointer walk.
Automatic Matching
No application changes: the engine inserts each request's token path into a tree; any future request sharing a prefix walks to the fork and prefill starts there.
TTFT Economics
A 4k-token system prompt at ~90 ms/1k tokens costs ~360 ms of prefill — every request. A cache hit deletes all of it; templated workloads see TTFT drop by half or more.
Eviction Policy
Cache blocks are reference-counted and evicted LRU. Hot system prompts survive; abandoned conversation branches decay. SGLang's RadixAttention made this the default.
At 40 ms/1k tokens and a 412-token shared prefix across 900 requests, the tree saves ~15 seconds of aggregate GPU prefill per minute of traffic — capacity you get for free.
- SGLang reports prefix hit rate as a first-class metric; fleets tune system-prompt design around it.
- Prefix caching composes with quantized KV and with PD disaggregation — the transfer shrinks to the unmatched tail.