The tiny model that outgrew its teacher

I gave a model 500 times smaller than a frontier LLM a broken Rust file and asked it to fix the compiler error. On the first try, it fixed nothing. Zero out of forty-five. This is the story of what I put around that model until it beat the much larger model that taught it, on cases neither had ever seen, running on four CPU cores with no GPU in sight.

Chapter OneThe 0% floor

Here is the confession up front: I did not expect this to work. I had absorbed the same folklore everyone in this space repeats, that sub-billion-parameter models are toys, fine for autocomplete and useless for anything that has to be correct. So when I dropped Qwen2.5-Coder-0.5B into my repair harness, pointed it at forty-five broken Rust snippets, and gave it a four-attempt budget per case, I expected a bad score. I did not expect a zero.

Not "solved a few, missed most." Zero out of forty-five. The model would look at a compiler error, produce something that vaguely resembled a fix, and my patch engine would reject every single one because the model could not even emit the edit format correctly. It was not wrong about the code. It never got far enough to be wrong about the code.

"A model that scores 0% and a model that scores 84% failed the exact same way once: neither could reliably say which lines to change. The difference is that one of them was never taught how."

That zero is the most important number in this whole project, because of what sat next to it. The same harness, same patch parser, same compiler verifier, handed to Gemini 3.6 Flash in a single shot, scored 84.4%. So the 0% was not a broken benchmark. The scaffolding worked fine. The tiny model was genuinely, completely failing at a task the loop made trivially checkable. The entire gap between 0 and 84 was headroom I could go get.

Frozen eval_v1, 45 in-distribution cases. Every run goes through the identical controller, SEARCH/REPLACE parser, and cargo check verifier. Latency is a non-goal here; the claim is accuracy.

Chapter TwoWhat the tiny-model debate gets wrong

We argue about the wrong thing. The whole "can small models code?" debate treats the model as the product, a lone brain that either knows the answer or does not. So we benchmark models one-shot, in isolation, and unsurprisingly the big ones win and the small ones look hopeless. Then we conclude the small ones are hopeless.

But almost nobody ships a raw model. They ship a system: a model wrapped in tools, retries, and something that checks the work. And a large class of real bugs, the kind that fill your terminal with red every day, are not open-ended reasoning problems at all. They are bounded and structurally identifiable. The compiler already told you exactly what is wrong and exactly where. The task is not "be brilliant." It is "make the correct small edit, in a valid format, and prove it compiled."

The bet behind nano-agent is simple: a large fraction of bounded, structurally identifiable failures can be repaired by a tiny model inside a verified loop, at a fraction of the cost and latency of a frontier call. The model is one component. The recipe, tool plus verifier plus patch engine plus bounded retry plus escalation, is the product.

Chapter ThreeThe compiler is the verifier

The recipe I built for Rust is called rust-repair, and the shape of it is deliberately dumb. There is no agent framework, no planner, no tool-selection reasoning. There is a loop, and the loop trusts cargo check to be the arbiter of truth.

Two design decisions carried most of the weight, and both came from watching tiny models fail.

SEARCH/REPLACE, not unified diff. My first instinct was to have the model emit a normal diff. That was the source of a lot of that original 0%. Sub-billion models mangle diff headers, miscount line numbers, and botch the @@ hunks. So the patch format is a plain "find this exact block, replace it with that block." No line arithmetic, no headers to corrupt. A grammar constraint forces valid JSON contract output on the way out, so the model physically cannot return a shape the parser rejects.

The context is the compiler's own words. The model does not get "here is a file, find the bug." It gets the rendered diagnostic, the file name, the primary error span, and the surrounding code. The compiler has already done the localization. The model just has to act on it.

Chapter FourAnatomy of one repair

Abstract loops are easy to nod along to and hard to believe. So here is one running. This walks a real category of case the way the controller sees it: a broken snippet, the model's proposal, the compiler's verdict, and the retry if the first attempt does not take. Step through it.

This is a re-enactment of the controller's real state machine, running locally in your browser. The compiler verdicts shown are the same ones cargo check returns for these error classes.

Chapter FiveThe night it beat its teacher

Getting the untrained 0% up to a real score meant teaching the model the one thing it could not do: emit a valid patch for a localized error. I did that with supervised fine-tuning on trajectories from a teacher model. And here is the deliberate, load-bearing choice of this whole project: the teacher is not a frontier model. It is Gemini 3.6 Flash, a cheap flash-class model that itself only scores in the middle of the pack.

Why handicap myself? Because the thesis I actually care about is not "a nano model can match GPT." It is "a nano model can beat the model that taught it." A distillation that surpasses its own source is a far more interesting claim than one that merely approaches a frontier it was directly trained on.

One SFT pass on 348 teacher trajectories took the in-distribution score from 0% to 86.7%, edging past the teacher's own 84.4% on that eval. Victory, I thought. Then I ran the honest test, and it humbled me.

So the real work began: a failure-driven curriculum. I found the skill classes the model kept failing, ten of them, from iterator-collect mismatches to trait-method insertion, and had the teacher generate fresh training cases for exactly those classes. Never the test cases themselves, always new instances of the same skill. The eval stayed frozen and untouched. This is the whole ballgame for honest evaluation, so I will say it plainly: you never train on the test; you train on new cases of the skill the test measures.

That curriculum lifted out-of-distribution accuracy from 44% to 60%. And 60 crossed a line that mattered: the teacher, Gemini Flash, scores 56% on those same held-out cases. The 0.5B had beaten the model that taught it, on cases neither had ever seen.

Then one more experiment, because I wanted to know what was capping the score: the model, or the teacher? I reran the same curriculum on the same ten failing classes, but this time distilled from a genuine frontier teacher (a model that solves 100% of the OOD set) instead of Gemini Flash. The out-of-distribution score went from 60% to 72%. A stronger teacher is a real lever on structural generalization. But notice the jump is bounded, and that boundary is the most honest part of the whole result.

A 0.5B model, quantized to a 600MB file, running on four ARM CPU cores with no GPU, scoring 72% on held-out structural repair. Sixteen points clear of the flash-class model that generated its training data. That is the headline, and it is a real, frozen-eval number.

Chapter SixUnder the hood: the choices that made it work

The story so far is the what. This is the why. Every decision below was a fork in the road where the obvious choice was wrong, and the reasons are the actual transferable content of this project. If you skim one section, make it this one.

Why a 0.5B, and why this exact base

The spec started with a rule I am glad I followed: do not open by deciding a parameter count is optimal. The plan was to sweep roughly 0.1B, 0.3B, 0.6B, and a 1B control, and let the task tell me the floor. In practice the 0.5B answered the question so cleanly that a full sweep would have been ceremony. Qwen2.5-Coder-0.5B-Instruct was the pick for three concrete reasons, not vibes:

• Code-pretrained. A code base model already knows Rust syntax and the shape of a compiler diagnostic. I am not spending my tiny parameter budget teaching it what a trait is; I am spending it on the repair convention.
• Instruct-tuned. It already follows a system prompt and emits structured output, which is the entire game when your contract is a JSON patch.
• It quantizes to a 600MB GGUF. At q8_0 the whole model fits in memory on a 4-core ARM box and serves patches in seconds on CPU. That is the deployment story, and it only exists at this size.

The point was never "0.5B is the magic number." It was to find the smallest model that, inside the loop, closes the gap. Bigger bases are a lever I am holding in reserve for the one place the 0.5B provably runs out of room, which is the next chapter.

The SFT setup, and why each knob is where it is

The training is deliberately unexotic. Full bf16 fine-tune, no LoRA, no QLoRA, on a single 24GB L4. A sub-1B model does not need parameter-efficient tricks to fit in 24GB, and full fine-tuning gives the cleanest signal, so QLoRA stayed in the code as an option for the 1.5B control I never needed. The settings that mattered:

The tell is in the numbers across rounds: eval token accuracy sat at ~0.95 for v2, v3, and v4, essentially flat, while OOD accuracy climbed 44 → 60 → 72. The fit did not change; the coverage did. That is the fingerprint of a data problem being solved with data, and it is why I could rule out "train harder" as the lever.

Curriculum: how the training data was chosen, then grown

The data pipeline is the part I would defend hardest, because it is what makes the numbers honest. Nothing is scraped and nothing is trusted on faith. Every single training example is generated, then confirmed by the real compiler before it is allowed in.

The first dataset (v2) was 348 verified teacher trajectories over twelve error classes generated by break_it.py, which makes single-site mutations: flip one token, drop one &, remove one mut. That taught the surgical-edit skill beautifully and overfit to it, which is exactly how the 86.7%-in-distribution model scored 44% on the held-out set.

The fix was not more of the same data. It was a failure-driven curriculum: I read the frozen eval's failures, identified the skill classes the model kept missing, and generated fresh, harder cases for exactly those classes with a second generator, structural_cases.py, that produces multi-line and structural bugs the single-site generator can never make. The rule I will repeat until it is boring:

"You never train on the test. You train on new cases of the skill the test measures. The eval stays frozen, hash-checked for zero overlap, forever."

That took v3 to 60%. For v4 I generated 250 brand-new cases, 25 in each of the ten still-failing classes, verified them disjoint from the eval by content hash, and only then distilled them. The dataset grew 348 → 489 → 739, and every increment was targeted at a measured, named failure, not volume for its own sake.

Teacher selection: the deliberate handicap

Here is the choice most people would get backwards. The obvious move is to distill from the strongest teacher you can afford. I did the opposite for the headline result and used Gemini Flash, a mid-tier model that itself only scores 56% on the held-out set, as the teacher of record. The reason is the whole thesis: a distillation that surpasses its own source is a genuinely interesting claim; one that merely approaches a frontier it trained directly on is not.

Then, as a separate and clearly-labeled experiment, I swapped in a frontier teacher (100% on the held-out set) to answer one question: is 60% capped by the model or by the teacher? That is the v4 jump to 72%. It is a teacher-strength ablation, not a change of thesis. The project's headline stays "a nano beats its flash-class teacher"; the frontier teacher only measures the ceiling.

Why no RL, yet

The spec left the door open for RL or a GRPO-style verifier-optimization loop, and I have a perfect reward signal sitting right there: the compiler returns a clean pass/fail on every attempt, which is exactly what RL wants. So why is there no RL in these results? Because SFT had not stopped working yet, and you do not reach for the heavier tool while the lighter one is still paying off.

Each curriculum round was still buying double-digit OOD gains (+16, then +12) at a cost of about fifty cents of GPU time. RL is more compute, more instability, and more moving parts, and spending it while plain SFT is still climbing would be premature optimization of the literal kind. The honest trigger for RL is a specific, measured plateau, and the next chapter shows exactly where that plateau finally appeared, and why it points at RL or a bigger base rather than another curriculum round.

Chapter SevenWhere a 0.5B runs out of room

Seventy-two percent is a win, not a solved problem, and the failures are more interesting than the successes. Seven cases still fail even after distilling from a teacher that solves all of them. And they cluster. Almost every remaining failure is in one family: insert a whole new trait method, or a whole missing import. Not "change this token," but "synthesize several correct lines and place them in the right scope."

That pattern tells me something the aggregate score hides. When the fix is a localized edit, the 0.5B has learned to generalize. When the fix requires composing a multi-line structural change, it hits a wall that more data from a better teacher did not move. That is the signature of a capacity limit, not a coverage gap. The next lever is not another curriculum round. It is reinforcement learning on that hard cluster, or a slightly larger base model, and that is a decision I want to make with data rather than vibes.

Chapter EightThe part most posts skip

Every result in this post is real, frozen, and reproducible from the repo. It is also narrow, and I would rather tell you the boundaries than let you assume they are wider than they are.

One model family, one seed. This is Qwen2.5-Coder-0.5B, trained once per configuration. I have not run multi-seed error bars or swept model families. The direction is robust across four training rounds, but the exact point estimates carry single-run variance. I have labeled them as measured figures, not statistical claims.

The eval sets are small. Forty-five in-distribution cases, twenty-five out-of-distribution. Small enough that a single case is worth 4 percentage points on the OOD set. They are hand-authored, frozen, and verified to have zero content overlap with training, but they are not a thousand-case benchmark. Treat them as a rigorous probe, not a leaderboard.

Latency is deliberately not a claim. The thesis is accuracy at low cost, running locally and privately. I did not tune or report throughput, and nothing here is a performance benchmark. The 600MB model runs comfortably on a CPU; that is the property I care about, not tokens per second.

Rust only, so far. The whole result is one language and one verifier, cargo check. The architecture is built to be language-agnostic (the controller does not know which language it repairs), but SQL and Terraform recipes are planned, not proven. Do not generalize a Rust result to your stack until the recipe exists.

It cost under six dollars. Total. All four training rounds on a rented GPU came to about $1.26; teacher and evaluation API calls were a few dollars more. That is not a disclaimer, it is part of the point: the entire experiment fit inside a coffee budget, which is exactly what makes the "don't pay frontier prices for routine repairs" argument concrete instead of theoretical.

Chapter NineWhat building it taught me

The verifier is the unlock, not the model. The single highest-leverage decision in this project was letting a free, perfect judge (the compiler) sit in the loop. It turned an unreliable proposer into a reliable system. Wherever you have a cheap oracle for correctness, you can probably shrink the model doing the proposing.

In-distribution scores lie, and they lie flatteringly. 86.7% felt like success until the frozen OOD set said 44%. If I had shipped on the in-distribution number I would have shipped a mirage. The held-out set, authored before training and never touched, is the only number I trust, and building it was the most valuable hour I spent.

A student can surpass its teacher, if you teach the right skills. The curriculum did not make the model a better mimic of Gemini Flash. It made it better than Gemini Flash at the specific classes I distilled, because the loop lets the student compound small correct edits the teacher would have one-shot and missed.

Know your capacity ceiling by its shape. The residual failures were not random, they were a family. That is how you tell "needs more data" from "needs a bigger model." Random failures say curriculum. Clustered failures that survive a perfect teacher say capacity.

Chapter TenWho this is for, and where it heads

A 0.5B model fixing Rust errors is a demo. The pattern underneath it is the point, and it reads very differently depending on where you sit. So here is the same result, told three ways.

If you are an ML engineer

The transferable asset here is not the model, it is the recipe shape: cheap proposer plus perfect verifier plus bounded retry plus escalation. Wherever your domain hands you a free oracle for correctness, you can probably run this play. A compiler is the cleanest example, but the same shape fits a linter, a type checker, a test suite, a schema validator, a SQL EXPLAIN, a Terraform plan, a JSON-schema check, or a units-of-measure verifier. The skill to build is not prompt-whispering a giant model; it is designing the loop and generating verified trajectories. That skill compounds across every task with a checkable output, and it is dramatically cheaper to iterate on than a frontier fine-tune. Fifty cents and fifteen minutes bought each round here.

If you are an enterprise executive

Three numbers matter to you, and they are all real: the routine repairs run on a 600MB model on a CPU you already own, they cost a fraction of a cent instead of a metered frontier API call, and your source code never leaves your infrastructure. That last one is the quiet headline. The dominant reason regulated teams cannot adopt AI coding assistance is that it means shipping proprietary or sensitive code to a third-party endpoint. A local nano model in a verified loop sidesteps the data-residency problem entirely for the common case, and escalates to an external model only for the hard minority, where you can apply policy. The frontier bill becomes a long-tail expense instead of a per-keystroke tax, and the compliance surface shrinks to the escalation path you control.

If you are a researcher

The interesting claim is not "small models are good." It is that a student can systematically surpass its own teacher when the teacher's skill is distilled into a verified loop the teacher itself never ran. The 56%-one-shot teacher produced a 72%-in-loop student. That reframes distillation from "compress a big model into a small one, losing a little" to "compress a specific skill into a small model, then let a mechanism amplify it past the source." There are clean open questions sitting right here: how far does the verified-loop amplification generalize across domains; where exactly is the capacity frontier for a given base size (my residual failures cluster suggests it is skill-shaped, not uniform); and whether RL against the same perfect reward closes the structural-insertion gap that more data would not. Every one of those is cheap to probe because the verifier makes the reward free and the frozen eval makes the claims honest.

Where this is heading

The broader industry is spending enormous effort making one very large model do everything, and that will keep mattering for open-ended reasoning. But a huge amount of day-to-day engineering is not open-ended. It is bounded, checkable, and repetitive, and that work does not need a genius on every call. It needs a competent specialist and a judge. The future I am betting on is a fleet of tiny, cheap, private, verifiable specialists, each owning one checkable domain, with the expensive general model held in reserve for the genuinely hard tail. This post is one specialist, in one language, with one verifier. The recipe is built to be language-agnostic on purpose, and SQL and Terraform recipes are next.

Chapter ElevenRun it yourself

The whole thing is open source under Apache-2.0: the controller, the SEARCH/REPLACE patch engine, the Rust verifier, the data generators, and every frozen eval JSON behind the numbers above. Nothing in this post is a claim you have to take on faith; each figure maps to a result file in the repo.

git clone https://github.com/mailtotanvir/nano-agent.git
cd nano-agent

# build the recipe
cargo build --release

# fix a broken crate with the tiny model (llama.cpp server on :8080)
./target/release/rust-repair --path ./broken-crate --backend llama \
    --model qwen2.5-coder-0.5b-instruct

# every run appends a full trajectory: diagnostics, each proposal,
# each applied patch, each re-verify, and the terminal outcome
cat trajectories.jsonl

The results narrative, the exact eval tables, and the reproducibility notes live in recipes/rust-repair/experiments/RESULTS.md. Break the eval, beat the score, or point the recipe at a language I have not tried yet. Every bug your compiler can name is a bug a tiny model in a verified loop might be able to fix, for a fraction of a cent, without your code ever leaving the room.