The loops lab needed a sandbox: a real Lima VM where agent-driven builds could run with their egress denied at the host, plus a doctor CLI that runs an adverse suite (prove the guest cannot reach out) and signs an attestation the unattended runner refuses to start without. Containment as a precondition, cryptographically attested. The build produced working code at every step, and its three best catches share one shape: a containment claim that looked airtight and was, on inspection, testing nothing at all.
Catch one: the passing test and the missing VM
The doctor’s first manual verification step was deliberately adversarial: run it against a Lima instance named definitely-does-not-exist and confirm it fails cleanly. It did not fail. It succeeded, and wrote a signed attestation saying overall: pass.
The mechanism is worth staring at. The guest runner only raised an error when a command failed to start; but limactl shell against a nonexistent instance starts fine as a process and merely exits nonzero, and exits-nonzero is exactly what a correctly-blocked adverse command does too. Every “try to reach the network” check was reporting containment-confirmed against a guest that did not exist. The fix checks the instance is actually running before every command, and the failure mode got its own named regression test. No pre-written test caught this; the plan’s own manual adverse step did.
Catch two: the load-bearing rule and the phantom interface
The provisioning script’s host-side firewall rule was described, in the script’s own comments, as the load-bearing control: a root-owned deny rule so the guest’s egress dies at the host. The pre-provisioning review (two AI reviewers run independently; their shared findings are a story already told elsewhere) surfaced the problem, and a dedicated research pass confirmed it: the script created the VM on Lima’s default networking, which exposes no distinct host-visible interface. The rule would have attached to nothing, silently. An operator could run the whole script, see zero errors, and have zero egress blocking, discoverable only if the adverse suite’s probes unexpectedly succeeded against a live guest.
The fix was a design change, not a patch: provision the VM on Lima’s shared vmnet network, which exposes a real bridge interface, and then, because that bridge is shared by every instance using it, filter by interface and the instance’s specific address, so the rule blocks this sandbox rather than every neighbour. A lock is a claim about a particular door; the fix had to first make the door exist, then make sure the lock did not bolt the whole corridor.
Catch three: the same bug, one layer up
Fixing the guest runner’s exits-nonzero ambiguity did not retire the bug class; the second review round found the identical shape twice more, one layer up. The adverse suite’s scratch-repo setup step discarded its own command’s return code, so a failed setup could masquerade as confirmed containment. And exit code 127, command-not-found, was being scored as a test outcome: a tool the guest did not even have installed yet would have produced a false containment-confirmed. Four independent review sources converged on that one. Both fixed; the suite now raises on setup failure and refuses to score 127 as evidence.
What next
The catches all happened before the first privileged provisioning step, which is the point: the discipline (adverse steps in the plan, independent review before provisioning, research passes on networking assumptions) was positioned upstream of the risk. The sandbox’s first real provisioning followed with the fixes in, and the attestation chain the whole phase exists for has a foundation that was tested by attempts to fool it.
What Next
The First Thing the Factory Was Allowed to Delete
Earlier in Building the Agentic Factory: the same instinct pointed at destruction instead of containment, and the scratch repo built specifically to be deleted wrongly.