When sparse parameters met physical memory

The systems premise: sparsity on constrained silicon

When I began investigating sparse mixture-of-experts models on commodity hardware, the promise looked almost embarrassingly simple: sparse activation should produce sparse resource use.

OLMoE activates eight of 64 experts per token at each of 16 MoE layers. Logically, that is exactly 12.5% of the expert weights. On an Oracle OCI A1 instance with four Arm Neoverse-N1 cores and 24 GiB of RAM, the arithmetic suggested an escape hatch: keep only selected experts hot, let native kernels do tensor work, and use BEAM supervision to coordinate requests and lifetimes.

Physical hardware does not honor a ratio in a model diagram. Page-fault policy, file-backed mappings, unions of experts touched by successive tokens, overlapping request lifetimes, and the time required to move a future expert into memory all intervene. Zincrom became an experiment in making those hidden costs visible.

The 0.270 mirage

The first dramatic result was wrong. Correcting it did more to shape Zincrom than any benchmark win.

An exploratory benchmark appeared to show the 8B dense model collapsing to an Operational Efficiency Factor of 0.270 under four workers. I initially read it as a memory-bandwidth wall. The denominator audit showed something less exciting and more important: four single-thread workers had been compared with four copies of a four-thread baseline on a machine with only four cores. The calculation silently assumed sixteen cores of compute.

I reran the suite with matched isolated controls and disjoint CPU pinning. The collapse disappeared. Contention efficiency sat tightly between 0.963 and 0.967 across the three evaluated models. Hardware counters were not collected, so these data do not prove a bandwidth mechanism; they do prove that the old normalization cannot support the collapse claim.

Splitting the graph without changing the answer

Once the baseline was honest, I could ask the real architectural question: can BEAM coordinate the sparse path without corrupting the math or consuming the compute window?

The native side keeps the GGUF tensors, evaluates the gate, selects top-K experts, and executes their gate/up/down projections. The BEAM side owns request admission, completion barriers, worker supervision, and eviction triggers. Model weights never travel through the Erlang heap, and the measured worker processes perform confirmation rather than expert arithmetic.

The reference callback captured normalized hidden states, expert IDs, gate weights, and outputs from llama.cpp commit 6e9007ae6. Three single token IDs—50279, 42, and 1000—matched at every one of 16 layers: 48 exact expert selections, zero gate-weight delta, and zero output delta. Causal batches 2, 4, 8, and 16 also matched exactly at all layers.

Batch 32 is the boundary. Expert IDs remained exact and weight delta stayed within its 1.0×10⁻⁶ gate, but layer 2 reached 1.1444×10⁻⁵ output delta against a 1.0×10⁻⁵ limit. I did not round it into success.

In the isolated 1,000-iteration harness, selected-expert computation measured 2,205.553 µs at p99. BEAM dispatch and confirmation measured 74 µs at p99—43.47 µs mean, 41 µs p50, and 53 µs p95. The p99 ratio was 3.36%, below the preregistered 10% limit.

The OS surprise: when 12.5% becomes 27.8%

OLMoE routes eight of 64 experts per token. In memory, logical weights are exactly 487,587,840 bytes out of 3,900,702,720 (12.50%). But physical memory allocation is governed by the OS page cache, not model architecture.

Under default file mapping on Linux, touching selected expert pages triggered kernel fault-around, mapping neighboring unselected pages into the page table. Measured via /proc/self/pagemap and /proc/self/smaps_rollup, PTE-present bytes reached 1,085,272,064 (27.7%) and peak RSS delta reached 1,089,163,264 bytes (27.8% of the 3.921 GB all-expert control), failing preregistered limits.

Applying MADV_RANDOM suppressed sequential readahead. Routed expert PTE presence dropped to 504,152,064 bytes (12.88%) and RSS delta fell to 506,138,624 bytes (12.98% of 3.899 GB), matching logical active sparsity within 0.48 percentage points.

A safe lifetime: synchronizing layer eviction

Kernel advice bounds one hidden state, but multi-token generation accumulates expert pages across successive tokens. Shared-model concurrency requires active lifecycle eviction.

Without eviction, an 8-token decode across 1, 2, and 4 concurrent requests caused peak RSS deltas of 2.439, 2.881, and 3.531 GB with expert PTE presence climbing to 60.97%, 71.32%, and 85.96%. Evicting after every token cut RSS to 1.146, 1.891, and 2.563 GB, but PTE presence remained excessive (27.96%, 46.17%, 61.43%) while elapsed time inflated to 5.2–5.9 s versus 0.8–1.0 s.

The solution is layer-completion eviction. Using a decoder callback, requests synchronize at layer ℓ. Once the final concurrent context finishes layer ℓ, its expert mappings are evicted before any request advances to layer ℓ+1.

Surviving churn: bounded memory under load

Synchronizing eviction at layer completion keeps memory strictly bounded across request counts and extended token horizons.

In staggered churn, layer eviction maintained bounded residency across all tested horizons:

Dynamic barrier membership completed 1,200 uneven-lifetime samples without deadlock. Layer eviction reduced peak RSS by over 90% while retaining 85.4%–90.8% of unmanaged throughput. Evidence ledger

When perfect knowledge fails

I supplied the prefetch harness with oracle knowledge of exact future-layer routes. That removed prediction error, but it did not make overlap reliable on four saturated cores.

Across twelve batch/layer-pair conditions, user-space page touch failed 7/12 latency gates, MADV_WILLNEED failed 11/12, and reserving one compute thread still failed 7/12. Strict CPU affinity reduced failures from 5/12 to 1/12 between runs, but a three-repeat batch-8 check still failed once at layer 14→15: overlap p99 was 15.533 ms against 14.733 ms sequential.

The network misses: streaming expert shards

If local prefetch competes with compute, I wondered whether a broker could move selected expert shards into the compute window. Redpanda gave a clean negative answer for this design.

Matched selected-expert compute took 7.2–43.4 ms. Fetching a real 31.1 MiB shard took 86.96–119.47 ms p99 through the native client and 95.14–138.68 ms through persistent brod. A 64 MiB control measured 260.95–269.44 ms. None of nine batch/layer conditions passed.

Policy is model-specific

Memory control became a scheduling problem: decide who enters, when a long waiter advances, and which recently used expert pages are worth retaining.

A four-active FIFO queue bounded memory but produced Jain fairness 0.739 and wait p99 71.584 s. Aging size-aware admission improved fairness to 0.901 and wait p99 to 65.016 s, with throughput rising from 18.64 to 19.26 t/s and peak RSS moving from 343 to 351 MB. Request p99 was 100.095 s under FIFO and 101.746 s under aging, so the fairness improvement was not a universal tail-latency win.

Hit rate alone did not rank the gains: Qwen had the lowest hit rate but the largest avoided fault cost per route and the largest measured improvement. These are model-specific results, not portable cache constants. Verified architectures use exact metadata; unknown architectures fall back to full layer eviction.

Context suspension and honest boundaries

Parked sequence states also consume anonymous memory. Reclaiming them forced a choice between fidelity, memory, and reconstruction cost.

Float16 suspension restored outputs exactly at all four evaluated checkpoints. Int8 changed the top expert in 2/4 cases, with mean absolute error 0.099–0.130, so it failed the fidelity gate. Always-suspend reduced peak RSS from 1.516 GB to 965.1 MB and retained 0.999 fairness, but throughput fell to 7.189 t/s versus 7.901 for LRU-4 and 7.609 for full eviction.

Longer quanta reduced reconstruction frequency but did not create a universal sub-1-GB bound. Under maximum-prefix stress, Q24 and Q28 reached 1.279 and 1.241 GB; with a 24-request queue they reached 1.059 and 1.056 GB.

Follow the evidence

The result I trust is not a slogan about sparse inference. It is a measured chain: correct the denominator, preserve native math, suppress fault-around, synchronize page lifetime, admit requests fairly, and close the paths that fail.

Local review and reproducibility