The shift from AI toolkit to ecosystem: lessons learned the hard way

The first post in this series ended with a platform that worked. This one starts with the question that kept me up after publishing it: working for whom? A gateway that works but logs your users' API keys in plaintext is not infrastructure. It's an incident report waiting for a date. Phase 5 was five milestones of proving my own platform would not hurt anyone, and it changed how I think about what "done" means.

Chapter OneThe demo trap

Here is the confession up front: when Phase 4 ended, I hit publish on a build log that implied my platform was production-shaped. It was not. It was demo-shaped. The difference hit me when I asked one question about my own gateway: where do the API keys go when things fail?

Think about what happens when an upstream provider rejects a request. The HTTP client throws. The error message travels up through layers of handling. Somewhere, someone logs it. Now ask: what does an upstream provider put in an error message when your Authorization header is malformed? Sometimes, the header itself. My logs were one bad provider day away from collecting every credential that flowed through them, in plaintext, in stdout, where nothing scrubs anything.

That question turned into Phase 5. Not "add features." Prove the platform cannot leak, cannot be abused, cannot double-bill, and survives its dependencies dying. Five milestones, in order: observability completion and guardrails, security hardening, chaos testing, billing, and launch packaging.

Chapter TwoThe log that could have cost me everything

The fix for the secret-leak problem is one module: secrets.ts in @tanvir1971/core. It is the smallest file in the platform and the one I would defend hardest in a design review. The idea: redaction happens at the last possible moment, in the logger itself, so that no amount of carelessness upstream can put a credential on disk.

Three layers, one engine:

The third layer is the one that earns its keep. An error message like "unauthorized for key gsk_abc123..." has no field name to trip the first layer. The value pattern catches it anyway, because a Groq key looks like a Groq key no matter which field you stuff it into. Sixteen tests in CI assert this. Try it yourself, this runs the same logic in your browser:

The engineering rule that makes this trustworthy: redaction is not a feature, it is a default. There is no opt-in flag. The logger scrubs everything, always, and the tests fail the build if a new credential shape escapes. My only regret is that I did not write this module in Phase 0. Every week it did not exist was a week the platform could silently hurt someone.

Chapter ThreeWriting my own attack plan

A threat model sounds like enterprise paperwork until you actually write one, and then it becomes the fastest architecture review you have ever done. I used STRIDE: six categories of bad thing, applied to every trust boundary in the platform. The exercise is humbling because the format forces a verdict per row: mitigated, with a named control, or accepted, with a stated reason. "I think it's probably fine" is not a verdict the table accepts.

Three findings surprised me, and all three are now documented rather than fixed, because "fixed" was not the right response:

JWT revocation is not real-time. A compromised tenant token lives until expiry. Fixing this means introducing a token introspection service, a new failure mode, new latency. For v1.0, I wrote it down as accepted, with the mitigation that tokens are short-lived. That sentence in a threat model is worth more than a half-built revocation endpoint.

Provider prompt caches are external. The input-caching feature sends prompt bodies to provider-managed caches. For a tenant with hypersensitive data, that is a data residency question I cannot answer from inside my platform. The mitigation is a config flag to disable native caching. The honest answer is in the document, not in my conscience.

Grafana is internal-network-only, and that is a feature I might remove by "helpfully" exposing it. The threat model row now stands guard: any exposure requires an SSO proxy first. Future-me wants to expose it for a demo. Documented-me has a table row that says no.

Chapter Three and a HalfThe engineering behind the proof

The story so far has been deliberately narrative. If you are an AI engineer or an ML platform lead, this interlude is the part you actually came for: what the hardening is made of, which libraries carry which guarantees, and where the residual risk deliberately still lives. Everything below maps to a file in the repository.

The control plane that enforces redaction

The logger is not a wrapper around console.log. packages/core-shared/src/telemetry ships createSafeLogger, scrubObject, and scrubSpanAttribute as one engine. The key architectural decision: scrubbing runs on both exits, stdout logs and OTel span attributes. A secret that survives the log path but rides a span attribute into ClickHouse is still a leak, so the span exporter goes through the same pattern engine. One threat-model row covers both sinks; one test suite asserts both.

The stack, and why each piece earns its place

Security posture: what is enforced, what is accepted

The CI security workflow runs three independent scanners with different failure classes: CodeQL (semantic queries for injection and unsafe deserialization), gitleaks (credential shapes in diffed history), Trivy (dependency CVEs and image layers). A threat-model acceptance never substitutes for a scanner and vice versa. The full matrix, with per-service STRIDE tables, lives in docs/security/; the platform-level summary:

Tech risk register, as a first-class artifact

The residual-risk section of THREAT-MODEL.md is the document I re-read most. Each entry names the risk, the reason it is accepted, and the trigger that reopens it:

For an AI engineer, the takeaway I would want: the interesting engineering is not the five capabilities, it is the discipline that every claim has a named test, every acceptance has a date, and every cross-language contract is a CI gate rather than a hope. For a CTO or CAIO, the takeaway is different: this is the checklist your platform team should be able to produce on demand. If they cannot name their threat model's residual risks and their reopen triggers, the platform is not hardened, it is unexamined.

Chapter FiveBreaking my own platform on purpose

Load testing tells you how fast your platform is. Chaos testing tells you whether it is honest when things go wrong. The distinction matters because the failure modes are opposite: load failures are loud, chaos failures are silent. A restart of Redis does not produce an error page. It produces whatever your code does when a cache returns nothing, which might be "serves traffic fine" or might be "every request 500s and the pager melts."

I wrote scripts/chaos.sh to find out which one I had. Three scenarios, each one a dependency being murdered and resurrected while the gateway is probed:

Each scenario is a real script, not a story: scripts/chaos.sh restarts the actual containers, probes the actual gateway, and exits non-zero on failure. The interactives above walk the same assertions the script makes. The uncomfortable truth this milestone produced: I did not know my platform's failure behavior until I forced it. Every silent assumption I had about degradation was unverified until a script made it lie to me or confess.

Chapter SixThe bug that would have double-billed everyone

Billing is the milestone I almost skipped as "not really infrastructure." It is actually the milestone with the least tolerance for error, because every other bug costs trust and this one costs money with a paper trail. The design in one sentence: a background sync reads per-tenant token usage from ClickHouse and pushes it to Stripe Metered Billing, and the entire module is gated behind a flag that defaults to off.

The interesting bug is the replay problem. Sync jobs fail and retry. Networks time out. If your sync is "add 15,000 tokens to the invoice," then a retry is "add 15,000 tokens again." Multiply by tenants and months, and you have invented a random tax on your most active users, discoverable only when they dispute an invoice.

The fix is Stripe's idempotency key, and the trick is what you anchor the key to. Mine is axiom-{tenant}-{period}. Same tenant, same period, replayed a hundred times: Stripe processes it once. But this only works if the usage action is set (total for the period) rather than increment (add to the running total), because set is naturally idempotent under the key while increment is not. Two decisions, coupled, either one wrong in isolation is a billing incident.

The tests encode the paranoia: a mapped tenant with zero usage is skipped, not zero-recorded; the flag-off build registers no routes at all, not 403s; the admin secret gates both endpoints; and a synthetic invoice preview returns the fixture, because the test suite must never touch real Stripe. Eight tests, all dry, all deterministic.

Chapter SevenFive repos, one tag

The last milestone is the one that turns a project into a platform: shipping. ADR 0007, written months earlier, made a promise the build had been quietly honoring. Develop in one workspace, structure every directory as a standalone repo, extract at release. Extraction day arrived, and the question was whether the promise held.

It held, because of a rule enforced the entire build: services never import each other, only @tanvir1971/core. That single constraint is what makes the split mechanical instead of surgical. The extraction script (scripts/extract-repos.sh) stages all five repos in seconds: copies the directory, injects shared CI and license, rewrites @tanvir1971/core from a workspace link to ^1.0.0, and prepares a fresh git history per repo. Cross-cutting assets (compose stacks, docs, ADRs) go to a sixth repo, axiom-meta.

Then the release pipeline. One tag, v1.0.0, and a workflow does the rest, in order, gated:

Two details worth stealing for your own release. First, the npm job refuses to publish unless the tag matches package.json exactly; version drift between the tag and the artifact is how phantom releases happen. Second, the multi-arch build builds amd64 natively and arm64 through QEMU, which is slow for the Python image, so buildx caching is scoped per service to keep the second tag push cheap.

Why this matters commercially, not just technically: extraction is what lets someone adopt the gateway without swallowing the whole platform. A team with their own RAG layer can pull one image. A startup can pin @tanvir1971/core and build their own services on the same contracts. Monoliths get evaluated. Packages get adopted.

Chapter EightHard-won lessons, half engineering half scar tissue

1. Ask "where does the secret go" before "does it scale." The single most valuable engineering hour of Phase 5 was tracing where an API key travels when a provider rejects a request. It found a leak path that 240 passing tests had walked past, because none of them asked the question.

2. Redaction at the logger beats redaction at the source. Ten code paths can leak a credential. One logger scrubs them all. Put the control where the exit is, not where the risk is, and enforce it with tests that fail the build.

3. A threat model is a decision record, not a document. The value is not the table. It is that every "probably fine" got replaced with either a named control or a written, dated acceptance. Future-you arguing with present-you needs the date.

4. Chaos scripts are cheaper than production incidents by about four orders of magnitude. Thirty minutes of writing a script that restarts Redis answered a question I had been carrying as unexamined faith for four months: does the gateway stay up? Now it is a CI-adjacent fact, not a belief.

5. Billing idempotency is two coupled decisions. Idempotency key anchored to the period, and set instead of increment. Either alone is half a fix. Test the replay path explicitly, because your sync job will be replayed whether you planned for it or not.

6. The monorepo promise only holds if you enforce the import rule from day one. Extraction worked because services never imported each other. One cross-service import in Phase 1 would have made this milestone a refactor. Architecture decisions are cheap when made early and devastating when discovered late.

Chapter NineThe part most posts skip

Credibility comes from what you admit, so here is the ledger for Phase 5.

Benchmark numbers are still empty. The k6 profiles exist, the thresholds are declared, the chaos scenarios run, but docs/benchmarks/BENCHMARKS-v1.0.md has a results table with no numbers in it, because I have not executed the full load run on a defined reference machine yet. It will only ever contain measured figures. Nothing in this post is a performance claim.

The Helm chart has not been rendered. helm is not installed in my environment, so the umbrella chart is syntax-checked but not template-rendered against a real cluster. First helm template run may find something. That is what untested deployables are: findable problems, stated plainly.

Billing is test-mode only, by design. Stripe integration runs on test keys behind a default-off flag. There is no production billing path yet, and the invoice preview is admin-gated and internal. Moving it past test-mode is a deliberate future decision, not a config flip.

Extraction is staged, not pushed. The script runs in dry-run mode and produces verified staging trees (dependency rewrite confirmed, CI injected, 210 files across five repos). The actual git push to the five new repositories is the step I am doing with eyes open, repo by repo.

And the platform is never done. JWT revocation, real-time threat intelligence in the security workflow, per-tenant ClickHouse row policies, and production billing all remain open. That is not a disclaimer. It is what the threat model's residual-risk section looks like when it is honest: hardening is not a milestone you complete, it is a posture you maintain, and the reopen triggers in the risk register above are how it stays maintainable instead of aspirational.