🔨 The Agent That Forges Its Own Tools: What Happened When I Stopped Adding Features and Let My Agent Grow a Library

Every agent framework ships with a fixed toolbox. FORGE ships with a forge. When it hits a capability gap, it doesn't fail; it writes a new tool, tests it in a sandbox, and registers it forever. Then I ran the experiment every skeptic would demand: does the growing library actually make it better?

Here is an uncomfortable question I kept avoiding for months: if you took away my agent's prompt engineering, its clever orchestration, its carefully tuned system message, what would actually be left?

The answer, for most agent frameworks I had built before, was: a very articulate caller of the same ten tools. Every framework in this space ships the same fixed toolbox: read file, write file, run command, search web. We argue about models and prompts while the toolset stays frozen. It's as if we built interns with perfect memory and never let them learn where anything is kept.

"A model gets smarter with every release. An agent's toolset gets smarter never, unless you build the mechanism that lets it grow itself."

FORGE is my attempt to build that mechanism: an autonomous coding agent whose defining feature is not what it can do today, but that when it discovers something it can't do, it forges a new tool. It writes it, tests it in a sandbox, and registers it permanently in a searchable skill library. Think Voyager's skill library, but pointed at general software and data tasks instead of Minecraft.

This is the story of building it and, more importantly, of the ablation experiment I ran to find out whether any of it is real.

⚙️ The Core Loop: Failure Is Just an Unwritten Skill

Most agent architectures treat task failure as an endpoint: retry, give up, or hallucinate confidence. FORGE treats failure as raw material. Every task runs through one loop:

A Skill is not a prompt snippet. It's a versioned Python module with a JSON schema, a docstring, unit tests, and provenance recording which task spawned it. Skills that pass their own tests become permanent, searchable citizens of the library, callable by this run, or by any future run, months later, on a problem nobody anticipated.

Skills that fail their tests are simply never registered. The library cannot rot with untested code, because untested code has no path inside.

🪄 What Today's Agent World Gets Wrong

FORGE exists because four failure modes keep repeating across agent frameworks:

That last row is the whole point of this post. Self-improvement claims are cheap. The interesting question is falsifiable: run the same benchmark twice, once with a library that persists and once with amnesia, and see if the library earns its keep.

🏆 The Money Experiment: Warm vs Cold

I built a 12-task benchmark suite across three domains (data wrangling, file/OS operations, API/parsing), each with a deterministic success checker. Before trusting any of those checkers, I validated all twelve against hand-written reference solutions run through the real sandbox (scripts/validate_checkers.py → 12/12 PASS). If the oracle is broken, the science is theater.

Then I ran the suite twice against a live LLM (Mistral codestral-latest, temperature 0.2): once warm (the skill library persists across tasks) and once cold (empty library, every run).

The warm arm's extra solve came from exactly the mechanism the hypothesis predicts: during synthesis, the planner sees "similar skills already in the library" and stands on accumulated shoulders instead of starting from zero. Cold-arm telemetry confirmed clean isolation: first retrievals returned zero hits, and the two arms' skill directories were completely disjoint. The delta wasn't leakage. It was learning.

And here is the part I'm proudest of, because almost nobody in this space prints it:

"An uncontrolled demo is marketing. A controlled ablation with printed limitations is science. I wanted the second thing, even though the first photographs better."

🔑 The Night the Infrastructure Fought Back

No blog post about autonomous agents should sound like the run was smooth. My first full ablation attempt nearly died in a storm of HTTP 429s.

The original provider's free tier starved the experiment mid-run: 123 of 205 LLM calls errored under rate-limit thrashing. The early numbers from that throttled run (warm 1/12 vs cold 0/12) were measuring quota exhaustion, not intelligence. Most people would hide those numbers. They're still in the repo's git history, relabeled honestly as an infrastructure stress test.

That failure forced two features that made the final experiment possible:

• Pacing and fallback chains. Providers are tried live in order, with automatic degradation: when the primary starved, FORGE walked the fallback ladder until it found codestral-latest, which then completed a full ablation arm with zero LLM errors.
• Honest telemetry. Throttled calls are counted separately from genuine quality rejections. In the failed run, roughly 16 of the errors were quota starvation versus about 5 real failures. That is a taxonomy you can only build if you refuse to blur the two.

The lesson generalizes far beyond this project: agent benchmarks without error taxonomies measure your API plan as much as your agent.

🧪 Under the Hood: Five Decisions That Define FORGE

💡 What Building It Taught Me

• The oracle matters more than the agent. Validating success checkers against reference solutions through the real sandbox was the least glamorous work and the highest-leverage. Without it, "58.3%" is a vibe.
• Self-improvement needs a bouncer. The test-gated registration step does more for long-term capability than any synthesis prompt engineering. Quality control is the growth strategy.
• Print your weaknesses. Six of twelve tasks fail consistently. Those failures, mostly the quality bar beating the model's skill, are documented as the highest-value next step, not buried. A results file that only contains wins is an ad.
• Infrastructure failures are findings too. The 429 storm produced pacing logic, fallback chains, and an error taxonomy the project would be poorer without. Run toward the flaky provider; it's teaching you what production will teach you later, for free.
• Voyager was right, just aimed wrong. The skill-library idea proved itself in Minecraft years ago. Generalize the domain and the concept survives intact. What changes is how hard it is to write good checkers.

🤝 Who Gets What

🚀 Where This Goes Next

The current delta of +8.3 percentage points across one seed and one model family is a directional result, and I've labeled it as such everywhere it appears. The next milestones are already tooled: multi-seed ablations, higher-temperature variance, a repeat-task suite to exercise reuse properly, and digging into the six consistent failures to understand whether the ceiling is the model or the checkers.

But the deeper bet is unchanged: the frameworks that win won't be the ones with the best launch-day toolbox. They'll be the ones with a forge. Every task an agent fails is either a dead end or a blueprint, and the difference between those two futures is a library that keeps receipts, gates itself with tests, and grows a little harder to beat every single day.

⚡ Try FORGE

The full loop, from synthesis through sandbox testing and registration to the warm-vs-cold ablation, is one clone away:

git clone https://github.com/mailtotanvir/forge.git
cd forge

# install (Python 3.11+)
uv venv .venv --python 3.11 && source .venv/bin/activate
uv pip install -e '.[dev]'

# configure any supported provider
cp forge.toml.example forge.toml   # add your key(s)

# run a task and watch skills get forged
forge run ap_001            # single task (add --cold for an empty library)
forge bench --seed 0        # full 12-task suite

# reproduce the ablation yourself
SEEDS=1 bash scripts/run_ablation.sh

Star the repo, break the test gate (please, I want to know how), and watch the library grow. Every skill it registers is one it will never have to learn twice.