The harness’s own pre-PR review convention was a cloud finder-subagent pass with no local equivalent, and the obvious next step was to build a local version: four review personas, checking correctness, architecture, simplicity, and efficiency. The hardware said otherwise almost immediately.
The actual problem: four personas, one model, no choice
The default hardware profile caps at one resident large model. Four personas on four distinct large models would mean four evict-and-reload cycles for every single review pass, a serialized, minutes-long tax on something meant to be a fast pre-commit check. So the panel became four system-prompt variations layered on one resident model instead, the same pattern the harness’s blind coding judge already used for deterministic scoring. Each persona is a rubric, not a separate mind.
That’s a real design compromise, made honestly and up front rather than discovered later. The question it immediately raises, does a panel of four hats on one head actually catch what four separate reviewers would catch, is exactly the question the eval work that followed had to answer with evidence, not hope.
What worked, then what got weird: recall was easy, precision wasn’t
Recall came first and looked good fast. A seeded four-issue fixture, one issue per persona angle, initially scored 0.75, three of four matched, traced to a clustering bug that kept only the longest finding text from a converged cluster and silently discarded the rest, in this case losing the one persona’s mention of the exact sibling function name a keyword match needed. Fixed to keep every distinct phrasing instead of picking one: recall reached 1.0, four for four. A second, larger eight-issue fixture on a disjoint codebase held at 0.875. Good numbers, by themselves reassuring.
Recall alone is a dangerously incomplete measure for a review panel, though, a panel that flags everything scores perfect recall and is useless. So the next fixture had zero seeded issues: genuinely clean code, nothing to find. The panel found things anyway. Three of four personas raised a problem on code with no problems: the efficiency persona flagged a pre-existing, unchanged loop as if the diff had introduced it; the architecture persona speculated about call sites the diff never showed; the correctness persona treated a narrower, intentional exception type as a regression.
The plateau: real improvement, then a real ceiling
Three concrete, traceable prompt gaps meant three targeted fixes: a shared “grounding discipline” restricting findings to the diff’s own changed lines, a rule against speculating on unseen code, and a strict severity rubric tied to actual runtime-behavior change, plus a mechanical rule that any hedge word in a finding (“may,” “might,” “could,” “likely,” “possibly”) forces its priority down to minor, which the precision check never counts as a false approval.
Live-verified across three tightening iterations against real Ollama: false approvals dropped 3/4 -> 2/3 -> 1/3, then held at 1/3 across two consecutive confirmation runs. Real, meaningful, and, this is the part worth taking seriously, it stopped. The one surviving false approval was itself hedge-worded (“may,” “it is unclear”) and still landed at important priority, in direct violation of an explicit, unconditional, mechanically-stated rule. Recall didn’t regress through any of this: both parity fixtures held steady, because the tightening acted on the priority field and recall-matching is priority-indifferent.
What next
The panel was, at this point, roughly halving its false-approval rate and holding there, real, evidence-backed, and still short of the design doc’s zero-false-approvals bar. The honest reading wasn’t “prompt tightening failed.” It was “prompt tightening had done what it could, and the remaining gap belongs to the model’s own instruction-following limits, not the prompt.” That reframing pointed toward a structural answer rather than a fourth tightening pass: verify what the model claims instead of trusting it to follow the rule.
What Next
Make the Reviewer Quote Its Evidence
Next in the Local Coding Harness arc: instead of a fifth prompt tightening pass, a second-pass verifier is added that has to quote the exact line it's objecting to, and gets checked, not trusted.