Shipping With a Watch, Not a Verdict

The change was small: a guard that stops a scaffolding script from registering a project unless both the repo and its vault entry sit inside the canonical estate. It went through five doubt-driven-development cycles before shipping. Every single one of the five found something the previous cycle had missed. The finding rate never reached zero, which is the reason this shipped with a written watch instead of a declared-safe verdict.

Five cycles, five real findings

Artifact

The five-cycle findings table

Cycle 1: guard ignored the vault path entirely; --no-register was defeated by a later --force-register; the test wrote to production registries. Cycle 2: --dry-run previewed the opposite decision from what it would actually do; a '..' path segment in a not-yet-existing tail failed open; --force-register was unproven through the argument parser. Cycle 3: a surviving mutant in the vault-prefix check; the new gate was the wrong shape and let the test suite pollute production. Cycle 4: the assert_ran helper had never actually checked anything; two mutations survived the entire suite undetected. Cycle 5: a symlink could fake canonicity in shipped code; assert_ran was STILL inert after cycle 4 claimed to have fixed it; the suite reported 42 passed, 0 failed with --force-register deleted entirely from the script.

The uncomfortable detail is not that five cycles found five sets of bugs. It’s that several of those defects were introduced by the fix for the previous cycle’s defect. The symlink fail-open in cycle 5 was created by cycle 3’s own “improvement” to the path-resolution logic; the simpler resolver it replaced had actually handled that case correctly. And two separate commit messages, across the five cycles, claimed a fix had been verified when it had not: cycle 4’s message said assert_ran was fixed and demonstrated; cycle 5 proved it was still inert. Both messages stand in the project’s history, uncorrected, because the record is append-only: the correction lives in the next cycle’s findings, not in an edit to the earlier claim.

What actually caught things

Self-inspection, re-reading the script, tracing the logic by hand, caught none of the headline findings across all five cycles. What did:

Mutation testing: deliberately breaking the shipped script in specific, plausible ways, then running the suite to see whether it noticed. This is what found the two silent survivors in cycle 4, and the fact that deleting --force-register outright, removing an entire code path, still left the suite reporting green.

Fresh-context reviewers, especially cross-model: every cycle’s most serious finding came from a reviewer or a mutation test, never from the author re-reading their own work a second time.

Checking a helper’s actual output, not just the verdict it produced: the .. path-collapse bug silently did nothing while the guard still returned the technically-correct answer for the wrong reason, a passing result that was accidentally right, which is indistinguishable from a genuinely correct one until someone checks the mechanism, not just the outcome.

Shipping with a watch instead of a verdict

The decision made explicit at the end of the fifth cycle: the registry guard ships, because every constructible mutation is now killed and both reviewers’ findings are closed. But it ships with a standing watch and named re-review triggers, not a declared-safe verdict, because green tests are weaker evidence here than usual, given that the finding rate never reached zero across five tries. The triggers are specific and written down rather than left to memory: anyone editing the guard’s core functions, any change to how the canonical roots are derived, or the guard’s own failure symptom appearing in the wild, a registry entry pointing at a path that doesn’t exist.

The part worth stealing

A test suite that has never caught anything is not evidence of correctness; it’s evidence you haven’t tried hard enough to make it fail. Before trusting a small guard script that “should be simple,” break it on purpose in a few plausible ways and confirm the suite actually notices, and if the finding rate across your review cycles never reaches zero, ship a watch instead of a verdict, and write down exactly what should trigger the next look.

What Next

Two Reviewers, One Exploit, Two Real Bugs

The flagship case for this whole arc: two independent AI reviewers on real PKI/CA code, both catches confirmed with a live exploit, before and after the fix.