Local LLMs and the Offline Factory

The first question everyone asks about local models is whether their machine can run one. It is the wrong question. The honest answer is “yes, probably, and that tells you almost nothing about whether you should.”

A laptop running a seven-billion-parameter model is a party trick until you can say which jobs it is actually good enough for. The interesting question is not capacity. It is assignment: can the factory decide which worker does which job, and back the decision with evidence rather than enthusiasm?

The question that changed

Choosing a model by vibes is the default, and it is seductive because the feedback loop lies to you. You try a local model on a task, it produces something plausible, and you conclude it is fine. You try a frontier model, it produces something slightly more polished, and you conclude it is better. Neither conclusion is grounded in anything except the last output you happened to see.

The shift was to stop asking “which model is better” and start asking “which model clears the bar for this specific task.” Those are different questions. “Better” is a vibe. “Clears the bar for summarising a session log into three bullet points” is something you can test.

Task profiles, not vibes

So the unit of decision became the task profile, not the model. A task profile names a job the factory actually does, the quality bar it has to clear, and a small set of eval cases with known-good answers.

Summarising a session log is one profile. Extracting decisions from a transcript is another. Classifying a project is another. Each gets its own bar, because a model that is plenty good at one can be embarrassing at the next. Once the profile exists, “should I use the local model here” becomes a test you run, not an argument you have with yourself.

The judge that could not be trusted yet

Running evals by hand does not scale, so the natural move is to have a model grade the outputs: an LLM as judge. This is where it gets recursive and slightly funny. To trust the local models, I wanted a judge. But a judge is also just a model producing plausible output, which is the exact thing I did not trust in the first place.

A judge that has not been checked against human verdicts is not a measurement instrument. It is another opinion, with a clipboard.

Calibrating the judge

So the judge had to clear a bar before it was allowed to set bars for anything else. I scored a set of outputs by hand, had the judge score the same set, and compared. The thing I cared about most was not raw agreement; it was false approvals, the cases where the judge passed something a human would reject. Those are the dangerous errors, the ones that quietly let bad output through.

On its eval set the judge reached full agreement with my verdicts and, critically, zero false approvals. That is the result that let me start using it. It is also a deliberately narrow claim: full agreement on one curated set, for these task profiles. It is calibration, not a certificate.

When to fall back to a hosted model

The point of all this is not local-model purism. It is knowing the boundary. Some profiles the local model clears comfortably, and those jobs can run offline, privately, for free. Some it does not, and for those the factory falls back to a hosted frontier model on purpose, with the reason recorded, not as a defeat. Model choice became an operating decision with an audit trail, which is a far less romantic thing than “I run everything locally” and a far more useful one.

The pattern: evidence over enthusiasm

The reusable lesson is small and unglamorous. Define the job before you judge the worker. Test against known-good cases, not impressions. And before you let one model grade another, make the grader prove it agrees with you on the cases where it matters.

This is lab work, and it should be read as lab work. The numbers are real and they are for one eval set, on one machine, for a handful of profiles. They are evidence that the approach is sound, not proof that any particular model is good. The day I forget that distinction is the day I am back to choosing by vibes, just with a spreadsheet to make it look rigorous.