The local coding lab runs a review panel on a local model: four reviewer personas over a diff, promoted only if it can match a cloud review board’s recall with zero false approvals. That promotion bar is the destination. This post is about the paperwork on the way there, because the lab’s eval log for those weeks reads like a diary of losing, on purpose, and that turned out to be the most valuable engineering artefact in the repo.
Three entries tell the story.
Entry one: mechanism fixed, number unchanged
Parity recall sat at 0.75, three of four seeded issues caught. A real bug was found in the harness, fixed, verified fixed. Recall: still 0.75. The honest sentence went in the log: the mechanism was repaired and the outcome did not move, which meant a second, independent bug, which live-verification then found. Recall went to 1.0 only after both.
The temptation at “mechanism fixed” is to write success and move on; the fix was real, after all. Separating mechanism verified from outcome achieved is what caught the second bug, because the gap between the two claims is exactly where a hidden problem lives.
Entry two: three of four false approvals, in writing
The promotion bar’s other half needed a control: a fixture of deliberately clean, well-written code, to measure whether the panel invents problems. First real run: false_approval_count = 3/4. The panel confidently flagged an exception-handling narrowing as a regression and raised concerns about code with no actual bugs, findings that sounded defensible and were, by the control’s construction, wrong.
The log recorded the number, the concrete shape of each wrong finding, and, in the assumptions section, the limit: a single real result on a single fixture, not repeated for variance; real evidence, not a large-sample guarantee. No softening, and no overclaiming in the other direction either.
Entry three: the win, and why it was provable
Weeks and several persona-prompt iterations later (one logged, characteristically, as “real, partial improvement”, because it was partial), a dedicated verifier persona met the promotion bar on every fixture for the first time. That sentence carried weight for one reason: every number behind it was already on the record. Recall’s path from 0.75 through the two-bug fix to 1.0; the false-approval count’s path down from 3/4; each step’s sample-size caveat. The win was a data point at the end of a documented curve, not a press release.
The part worth stealing
The cultural claim is the transferable one: in a one-person factory there is no boss to impress, and the eval log still drifted toward flattery until the format forced honesty. Whatever your equivalent of the promotion bar is, the discipline is identical: write the number you got, not the number after one more try; and when you fix something, say what the fix proved and what it conspicuously did not.
What Next
Calibrating an LLM Judge
Coming in this pillar's neighbourhood: the mechanics side of this story, how the judge itself gets calibrated rather than how its results get recorded.