I did not build a tiny model experiment. I built a miniature AI factory.
The original plan sounded modest: train a small model, try modern reinforcement learning, and expose it through an inference API. In practice, that sentence hid an entire AI product stack: data contracts, reward design, policy updates, reference policies, serving engines, routing, authentication, rate limits, usage accounting, artifact custody, and the uncomfortable discipline of asking whether the system learned anything at all.
The project stopped being a notebook when each choice created a new obligation
A training script can make a loss decline. A product-grade learning system must survive a more adversarial question: what happens when a reward improves, an engine returns a different completion shape, a caller exceeds a quota, a model route changes, an experiment is interrupted, or a result looks better than it deserves to?
The diagram is deliberately small, but the boundaries are real. Frozen split families made it harder to accidentally describe template familiarity as reasoning. A model registry separated public model names from a backend. The provider owned the customer-facing contract rather than allowing every engine to leak its own assumptions. Artifact copies on the laptop and existing OCI host meant a deleted GPU VM did not take the evidence with it.
I built five ways to push a policy, because the differences matter when the reward is expensive
“RL” is often used as if it describes one thing. It does not. Each family makes a different bargain between simplicity, sample efficiency, stability, memory, and the kind of feedback the system can actually obtain. I implemented the small versions not to claim a frontier reproduction, but to make those bargains concrete.
SFT establishes the language of the task
Supervised fine-tuning gives the policy an answer-shaped starting point. It is the cheapest path when verified demonstrations exist. It is also the control: without it, a reward-only method must spend scarce samples discovering basic format and behavior.
REINFORCE is honest about variance
Sample a completion, score it, and make rewarded actions more likely. The idea is clean. The tradeoff is noisy credit assignment: rare sparse rewards and long completions make variance expensive. In the stack it uses the same deterministic exact-match reward as every other path.
PPO trades theoretical purity for safer movement
PPO constrains how far the new policy can move from the policy that generated the data. That clipping is valuable when a good policy can be damaged by one oversized update. The cost is more moving parts: rollout bookkeeping, advantage estimates, and a reference behavior to compare against.
RLOO attacks variance without a learned critic
RLOO uses sibling samples as a baseline. Instead of fitting a separate value model, each outcome is compared with the other outcomes from the same prompt. It is attractive when a critic is undesirable or unstable, but it demands grouped samples and careful batch construction.
GRPO makes the group the critic
GRPO normalizes rewards inside a response group. For verifiable tasks, that can replace a large learned critic with relative evidence among candidates. It remains operationally demanding: generation count, global batch divisibility, reference behavior, KL control, and rollout cost are all part of the algorithm in practice.
KL is a choice, not a decoration
The stack measured k1 and k3 forms against the frozen base reference. Their estimates were close on 32 development prompts, but they are not identical by definition. More importantly, a label on a configuration is not a causal ablation unless the loss formula actually changes.
Click a situation. This is the practical decision tree I wanted before I started writing trainer code.
Select a situation to inspect the tradeoff.
Under the hood: what the implementations actually share
Qwen/Qwen2.5-0.5B-Instruct, PEFT LoRA adapters, frozen arithmetic template families, and conservative exact-match normalization in src/mini_stack/rewards/exact_match.py. The entrypoints are scripts/train_sft.py, train_reinforce.py, train_ppo.py, train_rloo.py, and train_grpo.py. The campaign contract gives each run an explicit seed, SFT adapter dependency, isolated output directory, and metrics path. That common substrate made method differences easier to inspect and made campaign artifacts collectable.The model is only half the machine: an adapter becomes a service through a chain of compromises
Training an adapter is not the same as making it useful to another application. The adapter needed merging for engine compatibility, a registry entry, a serving runtime, a caller-facing schema, operational limits, and a way to explain which backend handled a request. This is where a model experiment begins to look like an inference product.
The provider surface is intentionally narrow but real: /health, /v1/models, /v1/completions, /v1/chat/completions, /v1/responses, /usage, and /metrics. The application owns bearer-key validation, deterministic routing, rate limits, token accounting, streaming completion behavior, and Prometheus-style metrics. That separation is not ceremony. It stops a client integration from being coupled to one engine’s server, one vendor’s model label, or one local deployment shape.
| Concern | Implementation boundary | Why it matters in a provider ecosystem |
|---|---|---|
| Model identity | Versioned model registry and deterministic router | Clients call a stable public name while the backend can change. |
| Access | Hashed bearer key check | Authentication must happen before costly inference work. |
| Cost and abuse | Per-identity token bucket and usage ledger | A provider needs a way to say who consumed what and when. |
| Compatibility | OpenAI-shaped chat, responses, completions, streaming | Existing clients can integrate without bespoke engine code. |
| Backend choice | Async protocol boundary for vLLM, SGLang, and remote OpenAI-compatible backends | Serving behavior becomes a replaceable implementation choice. |
42, 10 prompt tokens, 1 completion token, and 1578.84 ms recorded latency. That does not prove universal reliability. It proves the provider boundary accepted a live remote backend without exposing a provider-specific contract to the caller.The engine comparison taught a less glamorous lesson: a benchmark is a protocol, not a number
vLLM and SGLang were both exercised on the same L4 with the same merged SFT adapter. The smoke run returned 139.72 ms in vLLM and 108.58 ms in SGLang for one warmed prompt. It would have been easy to call that a winner. It would also have been wrong.
139.72 ms
16 prompt tokens, 9 completion tokens. The returned completion included 42. This is evidence that the serving path worked for that request, not a global throughput claim.
108.58 ms
16 prompt tokens, 16 completion tokens. The completion continued beyond the desired response and reached the output cap. That makes the one-request comparison non-equivalent.
The most useful part of the stack was the boundary I could test without a GPU
The GPU was necessary for post-training and engine exercise. It was not necessary for every important engineering decision. Schemas, canonical serialization, configuration hashes, campaign state transitions, routing, authentication, rate limits, usage accounting, and the OpenAI-compatible contract all ran locally. That separation made the expensive campaign smaller and made the GPU hours evidence-producing rather than debugging time.
First, make the experiment resumable
A campaign state machine records planned, running, completed, and failed stages. The publication orchestrator creates isolated output and evaluation paths for every seed and method. This is what turns “I ran something” into a recoverable execution record.
Then, preserve the evidence before deleting the machine
Each campaign archive was copied to the laptop and existing OCI storage. SHA-256 values matched before the L4 VM was deleted. The final metric-complete archive hash was 1383a5eb7c3757bac825604d9712e4ed783f4ee24df173868d2c6dbda25adf5f.
Finally, separate completion from readiness
The Definition of Done matrix marks custom provider behavior, routing, streaming, usage, two-copy artifacts, and model serving as complete where evidence exists. It preserves partial status for the integrated end-to-end GCP provider campaign and for KL ablation strength. That distinction is how a project stays credible when it is still growing.
The result that refused to flatter me
The final metric-complete campaign ran seeds 13, 42, and 97 through SFT, REINFORCE, PPO, RLOO, GRPO, and two GRPO estimator labels. Every one of the 21 cells completed. Every saved adapter was then evaluated on the held-out division-template family. Every result was 4 correct answers out of 128.
| Method | Seed 13 | Seed 42 | Seed 97 | Mean |
|---|---|---|---|---|
| SFT | 4/128 | 4/128 | 4/128 | 0.03125 |
| REINFORCE | 4/128 | 4/128 | 4/128 | 0.03125 |
| PPO | 4/128 | 4/128 | 4/128 | 0.03125 |
| RLOO | 4/128 | 4/128 | 4/128 | 0.03125 |
| GRPO | 4/128 | 4/128 | 4/128 | 0.03125 |
| GRPO k1 and k3 labels | 4/128 | 4/128 | 4/128 | 0.03125 |
GRPO still produced a meaningful internal training signal. One completed cell recorded mean rollout reward 0.79167 and KL 0.37643, then scored 4/128 on the held-out task. That is the central learning: a reward-bearing optimization loop can be internally alive without producing the generalization outcome we care about.
Fine print I refuse to hide
What I would carry into a real AI team
Pick the learning loop that matches your feedback, not the algorithm with the best reputation
Verified answers, preferences, process rewards, and long agent trajectories produce different credit-assignment problems. The reward source should decide the post-training design. Build a clear SFT control and a hard held-out boundary before spending on rollout sophistication.
An inference engine is a component. A provider is a contract.
vLLM, SGLang, and a cloud backend are execution choices. Authentication, routing, rate limits, usage, model identity, streaming semantics, and evidence are the product boundary. Put that boundary in one testable place.
The cost of a run is not the bill. It is the bill plus what you cannot explain afterward.
An on-demand L4 can be affordable when the local work is ready, output paths are isolated, campaign evidence is explicit, artifacts are copied twice, and the machine is deleted immediately. A cheap run with no provenance is expensive learning.
Negative transfer can be the finding when the measurement chain is intact
The value here is not a leaderboard claim. It is a compact system that keeps reward traces, held-out scoring, serving limits, configuration boundaries, and artifact custody attached to the same story.