I built an operating system for AI.
It humbled me.

This is not a post about prompts. It's about what happens after the prompt works: the outage you didn't plan for, the bill nobody can explain, and the question "why did it say that?" when your whole career is on the line. I rebuilt the layer under every AI app I've ever shipped. Here's what happened, including the bug that made me look like a fool to my own test suite.

Chapter OneThe week it all broke

Let me start with the moment my relationship with AI engineering changed. Not a success story. An outage.

A provider I depended on had a partial failure. Nothing dramatic, just enough degraded responses to poison my app. And here's the thing that kept me staring at the ceiling that night: there was no seam anywhere in my code where a second provider could have existed. The SDK call sat directly in the request path like a load-bearing wall. To add a fallback, I would have had to rebuild the room.

That same week, two more walls came down. A stakeholder asked me which prompt version and which model produced a specific bad output, and roughly what it cost us. I opened my editor, scrolled through prompts pasted between functions like comments, and realized the honest answer was: nobody knows, and nothing was designed to know.

And then I watched an agent burn through tokens retrying a failing tool call. No budget. No cap. No event history. It failed silently until someone noticed the bill.

The models were fine that week. Every single failure was mine. They were plumbing failures, accounting failures, custody failures. The boring kind of failure that software engineering solved decades ago for databases and message queues. The AI industry just hadn't gotten the memo yet.

So I made an unreasonable decision. I would stop assembling features and build the substrate properly: a gateway, a knowledge pipeline, an agent engine, a control plane, and a shared contract library. Five packages. Production rules. If a claim couldn't be proven by a test, it didn't go in the README.

This is the story of building it, told honestly, including the parts where the system outsmarted me.

Chapter TwoOne rule before line one

Before writing code, I wrote down how the project could fail. Not runtime failures. Project failures: phases that end when enthusiasm ends, scope that grows because nothing stops it, quality that decays because "done" means different things on different days.

The countermeasure was one rule, enforced ruthlessly:

A phase doesn't exit when its code exists. It exits when its evidence exists.

Every phase got exit criteria written before implementation, each mapped to named test files. Gateway couldn't close until three providers failed over live. Agent runtime couldn't close until I killed a worker mid-run and proved it resumed without repeating itself. RAG couldn't close until a red-team suite failed to cross tenant lines.

The rule changed my relationship with testing. Tests stopped being the tax you pay after building and became the proof-of-reality layer the whole project stands on. When I later claimed "one trace follows a request across four services," that wasn't marketing copy. There is a test file that asserts the actual bytes of the trace header. Hold that thought, because that test is also where this story gets embarrassing.

Execution itself was agentic. I ran an AI dev-engine as a pair, but the division of labor mattered more than the tooling: the agents wrote code fast; the criteria decided what counted. On the days everything looked finished, the criteria said otherwise, and the criteria were right. More than once.

Chapter ThreeHiring a bouncer for my API keys

Every AI app has a moment where a user's words become someone else's API call. That moment is where money leaks, outages begin, and costs become unexplainable. So that's where the build started: a gateway, a service whose entire job is to stand between your application and the model providers and impose order.

The pitch sounds simple. Proxy chat completions across Groq, Mistral, Gemini, SiliconFlow, NVIDIA NIM, all speaking OpenAI's wire format. The devil is in what "impose order" means:

My favorite piece hides in the economics. Most teams treat caching as an optimization footnote. Here it's a first-class subsystem with its own dashboard, its own hit-rate metrics, and its own honest limitation recorded in the docs: the exact-match tier is process-local until the multi-worker scale-out lands. Saying what isn't done is part of the feature.

Chapter FourTeaching software to remember (without leaking)

Retrieval looks easy. Embed documents, search vectors, return chunks. Every tutorial ends there. Production begins where the tutorials end, with three questions nobody asks until the audit:

Whose memory is this?

In a multi-tenant system, the scariest bug class is silent crossover: customer A's document fragment surfacing in customer B's answer. My rule became structural: tenant scope comes from verified credentials and is enforced by the data layer itself. A caller cannot pass filters that cross tenants, because the mandatory claim filter is not the caller's to modify. Then I hired an adversary. Nine red-team tests try forged signatures, mismatched credentials, crafted filter payloads, cache probing across boundaries. All of them fail closed. That suite passing matters more to me than any benchmark score in this post.

How do we answer the same question twice, for free?

A two-tier cache. Exact matches short-circuit entirely. Paraphrases hit a similarity tier backed by Qdrant. The interesting part isn't the hit, it's proving the hit is free: the test asserts zero embedding-provider calls on the hit path via transport mocking. Anyone can claim a semantic cache saves money. This one has a test that counts API calls.

Is retrieval actually good?

"Feels accurate" is not an engineering property. A golden set of topics drives a recall@10 regression gate wired into CI, scoring every build. Currently 1.00 against a threshold of 0.9. If a chunker change quietly degrades retrieval, CI turns red before any human notices something feels off. Chunkers themselves are property-tested for lossless reassembly, because if chunking silently drops characters, nothing downstream, including your evals, can detect it.

Chapter FiveThe agent that could not be killed

Agents are where demos go to die. In a demo, the loop completes. In production, the worker gets OOM-killed mid-plan, the tool hangs forever, the webhook receiver is down for maintenance, and someone asks why the agent "did that." This package exists to answer those moments.

Kill it. It comes back.

Agent execution is event-sourced: every step, plan, tool call, observation, persisted as it happens. The signature test literally kills the worker mid-run and brings up a fresh one, which replays from the last event and finishes the job without re-running the planner or duplicating a single step. Underneath sits plain BullMQ rather than Temporal, a deliberate bet: event sourcing bought most of the durability at a fraction of the operational cost. The deferral is documented with a revival trigger, not forgotten.

Untrusted code meets a locked room

Tools execute inside an isolated VM with a serialized bridge only. No host handles cross the boundary. CPU-time caps kill infinite loops. Heap caps kill bombs. Module loading, network access, filesystem egress: blocked by construction, and asserted by a red-team suite I genuinely enjoyed writing. Every escape I could imagine, turned into a test that must fail.

Webhooks: physics vs promises

Delivery is at-least-once; that's physics and you don't fight physics. Observation is exactly-once: receivers dedupe by HMAC signature, timestamp replay protection, and event ID. The integration test delivers the same event three times and asserts it was processed once, with tampered deliveries rejected before recording anything. Failed deliveries land in a dead-letter queue with a replay CLI, so "temporarily down" never means "permanently lost."

Chapter SixThe witness stand

Four services produce behavior. The fifth produces evidence. I think of the ops plane as the witness stand where the rest of the system explains itself, and it permanently changed how I ask questions about AI systems:

The experiment engine carries my favorite design constraint in the whole project: the gateway resolves assignments locally from polled rules, so when the control plane goes down, requests degrade to "everyone gets control" rather than "nobody gets completions." Compare that to the security paths, sandbox escapes, forged webhooks, cross-tenant reads, which fail closed by design. Knowing which paths may fail open and which may not is half of reliability.

The stats are honest too: seeded Monte-Carlo win probabilities from wide posteriors, arms with no data can't win by default. Small-n experiments produce humble conclusions, on purpose.

Chapter SevenFive repos, one handshake

The smallest package carries the most architectural weight. Types for requests, chunks, usage, errors, tenants. Zod schemas mirroring the environment contract so startup fails fast on misconfiguration. HMAC signing shared identically by webhooks and inter-service auth. The telemetry module every service instruments with. Protobuf definitions gating breaking changes behind major versions.

Why does this matter? Because drift between services is a contracts problem before it's a deployment problem. Without the shared package, service A's "validation_failed" becomes service B's mystery 500. With it, breaking changes require a major version bump in public, with a changelog. It felt like overhead for about a week. Then it felt like gravity.

@tanvir1971/core
├── types/       requests · chunks · usage · tenants · errors
├── config/      Zod schemas (startup fails fast, loudly)
├── crypto/      signPayload / verifySignature (one format everywhere)
├── telemetry/   OTel SDK + Gen-AI span conventions   ← plot twist lives here
└── proto/       axiom/v1 (the source of truth)

Some decisions I sweated over, recorded so future-me can judge present-me:

Chapter Seven and a HalfLift the hood

The story so far has been deliberately narrative. This interlude is for the reader who now wants to see the machine: what runs where, what talks to what, and how one request actually moves through all of it.

Everything that runs, on one page

Versions are pinned in each package manifest; the constraint that shaped this stack is that only @tanvir1971/core is published, so every shared dependency upgrade is a single semver-gated decision.

A few deliberate choices hide in this picture. Every arrow that crosses a service boundary carries W3C traceparent, which is why a single trace ID can reconstruct an entire journey later. Every store has exactly one job: ClickHouse for immutable telemetry rows, Postgres for stateful artifacts (prompts, run events, experiment records), Qdrant for vectors, Redis for anything ephemeral. No store is doing two jobs; no service reaches past another to touch a store it doesn't own.

Anatomy of one request

The whole proxy tax lives in the first few milliseconds of bookkeeping. Everything after routing is the upstream model's own latency, untouched.

Where the hard guarantees actually live

That's the machine. Now back to the story, because the best part, the part where the machine fooled me, comes next.

Chapter EightThe bug that left no trace

Every long build has one bug that humbles you. Mine arrived in the final week, wearing a green checkmark.

The last open deliverable was correlation: prove one trace ID follows a request from client through gateway to provider. Traces were flowing. Spans were landing in ClickHouse. Dashboards rendered. Everything looked observably, verifiably fine.

Then I wrote the proof test: capture the outbound request at a mock provider and assert it carries a valid W3C traceparent descended from the inbound one.

expect(outbound.traceparent).not.toBe("");
// → received: ""

Empty. Not malformed. Not missing sometimes. Empty.

The root cause read like a riddle. OpenTelemetry ships a default propagator called NoopTextMapPropagator. Its job is to do nothing, gracefully. Registering a tracer provider, which I had done, does not register a propagator. Nobody tells you this. No warning fires. Every "inject" call I made was injecting context into a void, and the void accepted it politely.

And there was a second layer to the embarrassment: even with a real propagator, my gateway injected from an empty ambient context. It never extracted the inbound headers in the first place. So even "fixed," traces would have started chains from scratch, severing the very correlation the whole exercise existed to create.

The fix went in at both layers. The shared telemetry initializer now registers the W3C propagator explicitly, so every service inherits correct behavior by existing. The gateway extracts inbound context and opens a server span per request, so clients that send no tracing headers still get a valid chain. The test turned green and stayed in the suite as permanent evidence. The bug that left no trace now leaves a very specific one.

Honorable mentions from the same campaign: a not-found handler shadowed by parameterized routes (caught by scaffold test), colons in job IDs breaking Redis cluster slotting (fixed, commented, unfixed-able again), and a stray local process squatting on port 4000 during verification, immortalized as an honest port deviation in the tracker rather than silently papered over.

Chapter NineWhat I'd tell you over coffee

Strip away the code, and here is what four months actually taught me.

Evidence is a culture, not a phase. The exit-criteria rule felt bureaucratic for exactly three days. Then it caught a failing assumption I would have shipped proudly. Definition-of-done written first, testable, arguing back: cheapest senior engineer available.

Adversarial design is cheaper than adversarial incidents. The security suites, sandbox escapes, tenancy attacks, webhook forgery, shipped cleanest of anything in the project. Not because I'm clever. Because they were hostile from the first commit instead of bolted on after the first scare.

Cost is a first-class requirement. Metering with reconciliation deltas reshaped design decisions far from billing. Cache-hit paths proven to avoid embedding calls. Prompt-cache discounts surfaced instead of silently absorbed. If you can't attribute spend per tenant per feature, you're not running a product, you're sponsoring one.

Prompts are artifacts, not strings. Immutable versions, validated variables, promotion gates. Once "which prompt produced this?" has a real answer, eval scores mean something, rollbacks become possible, and prompt changes stop being folklore.

Defer loudly. Temporal, Celery, Unstructured, Stripe GA: every cut is written down with the condition that revives it. "We'll revisit if X happens" is strategy. Silence is just debt with better branding.

And document the deviations. The port conflict, the swapped cache backend, the forward-looking metrics scrapes. Readers trust documentation that admits limits. I trust engineers who write them.

What different readers should take away

The asides below are aimed at different readers on purpose. If you build products on AI, the five questions are your procurement checklist. If you write AI systems, the patterns are the transferable seams and gates. If you own the budget, the CTO list is what to demand in a design review before writing a check.

If you're an AI engineer or ML expert

If you're a CTO or CAIO

If you build AI systems yourself

Chapter TenThe part most posts skip

Credibility comes from what you admit, so here's the ledger.

Not load-proven at scale. Formal k6 profiles, 1k concurrent streams, webhook storms, are Phase 5. Today's performance claims come from targeted benchmarks with stated methodology, not sustained production traffic.

You operate it. This is self-hosted infrastructure. Single-node compose by default; retention, tuning, and pager duty are yours. The Prometheus scrape targets declared in this repo were written ahead of the services exposing /metrics exporters, and the dashboards say so plainly.

Some plan features shipped as deviations. Background tasks replaced Celery workers. Native parsers replaced Unstructured. Eval scorers run in-process rather than as subprocesses. Each recorded with rationale. Deviation logs are how a project stays honest with its own roadmap.

And it's not done. Guardrails middleware, a full docs site, Helm charts, and v1.0 tags remained at the end of Phase 4. All five landed in the sequel, and the honest close-out of every one of them, including what stayed accepted rather than fixed, is in The shift from AI toolkit to ecosystem.