The honest record of small local models doing real work in this factory is mostly a record of declines. The sibling coding lab watched a 7B model produce the identical malformed diff twice in a row, unmoved by the corrective error message it had just been shown. If your mental benchmark for “useful” is does the frontier-model job locally, a 3B model is a rounding error with a fan noise.
The engine gateway’s prompt-compiler phase found the job the small model is actually good at, and it is upstream of all of that. gateway-prep hands the raw human prompt to llama3.2:3b, running locally through Ollama, and asks it for structure, not brilliance: classify the prompt into one of the policy’s task types, draft a short structured brief (goal, constraints, acceptance criteria), and pick the context files worth attaching from the repo. The output is a valid, typed task spec. The expensive model, whichever one the router picks, starts from a prepared brief instead of a vague sentence.
Why this split works when the other one doesn’t
Every failed local-model experiment here had the same shape: reasoning-heavy execution with a correctness bar (write the patch, fix the bug). Prep is the opposite shape. Classification against a short list of task types, templating a brief, and shortlisting context files are structure-and-selection problems: small vocabulary, small output, wrong answers cheap to catch. A bad patch breaks a build; a mediocre brief just gets improved by the model executing it.
And the cost asymmetry runs the right way. Prep on a 3B model is effectively free and private, so it can run on every task, including the ones that end up dispatched to a human handoff. When the live path was validated, the small model produced a genuinely usable structured brief and chose the two context files a human would have chosen: the routing policy and the index doc, for a task about routing.
The guardrails that make it safe to trust
A model this small earns its place only because the wrapper never lets it improvise. Its task-type output is clamped to the policy’s known types; its privacy label is clamped to the three real levels; the human’s original prompt is preserved intact inside the prepped spec, so nothing the small model misunderstood is lost. There is a fully deterministic fallback path (--no-llm) that classifies by keyword and extracts context from file mentions, so the pipeline works offline and the LLM path is an upgrade, never a dependency. And the load-bearing invariant, pinned by tests: a prepped spec routes to exactly the same engine the original prompt would have. Prep improves the brief; it is structurally incapable of smuggling a task past the router.
The part worth stealing
If you have an idle local model and a subscription to a frontier one, stop auditioning the local model for the lead role. Put it in front: classify, template, shortlist, summarise inputs, so every expensive call starts warm. The division of labour that finally worked here was not “local does easy tasks, cloud does hard ones”. It was “local does structure, cloud does judgment”, on the same task, in sequence.
What Next
The Indexer Nobody Called
Also in Local LLM Lab Notes: before trusting any component with a job, check that something actually calls it. The sibling lab learned that one the quiet way.