A multi-agent colony simulation compared two coordination regimes head to head, live, with real models: peer-delegation, where agents propose their own actions, against central-orchestrator, where one agent plans everyone’s turn. The result looked clean. Peer-delegation weathered the storm. Central-orchestrator starved the colony. It read like a real finding about coordination architecture, until an independent review asked one question the original comparison had never checked: was this actually the same cast on both sides?
The confound hiding in plain sight
It wasn’t. The peer-delegation run everyone had been citing as the successful baseline had a scripted, not live, Quarter: the very agent whose planning behavior central-orchestrator concentrates authority into. The comparison that looked like “two coordination regimes, same live models, different results” was actually “one regime with a live planner against one regime with a scripted one.” A live model’s planning failures were being compared against a script that could not fail the same way.
Finding this did not require new data. It required checking a field that was already on disk (the run log’s own generated_by record) against the claim being made about it. Nobody had verified that the two arms were actually matched before treating the comparison as decisive.
The rerun that reversed the finding
The corrected experiment ran both regimes with all three agents genuinely live, no scripts on either side. The result: both runs failed to weather the storm. Central-orchestrator’s p-value came in at 0.4; peer-delegation’s matched-cast rerun came in at 0.55, milder, but still a failure. The original headline, “peer-delegation works, central-orchestrator doesn’t,” did not survive contact with a fair comparison. What the live evidence actually supported was narrower and less flattering: no live model had yet been trusted with the planner’s job, in either regime.
Digging into why surfaced the root cause directly in the code, not by inference: the context-building function for peer-mode explicitly left Quarter’s planning-guidance beliefs untouched, by design, because they had been built for the central-orchestrator regime and never ported back. Quarter’s live peer-mode planning had never been prompted for at all: not tested and found lacking, simply never built. The original “decisive” result was, underneath the confound, actually a story about an untested code path being compared against a tested one and calling the difference an architecture finding.
Fixing the gap, then re-running again
The team built the missing prompting for peer-delegation’s Quarter only (five guidance-belief conclusions mirroring the fixes already made for the other two agents) and re-ran the same peer-delegation live trial pair. Both runs succeeded this time, storm weathered, matching the stub baseline’s own final numbers exactly. Central-orchestrator was not touched in this pass: its live prompting gap remains open, and its earlier result, storm_weathered=false, p=0.4, was never re-tested against a fix, because no fix for that regime was built. The asymmetry the very first live comparison suggested is, on this narrow evidence, real again: peer-delegation now has a demonstrated live fix behind its result; central-orchestrator’s equivalent fix was never attempted. An independent cross-model review of the Quarter fix, run before it was called done, found two further real bugs: the new guidance actively taught Quarter to emit an allocation decision whose underlying handler had the identical unvalidated-crash exposure the original context-building bug had (fixed by requiring all three of its fields, none of which has a sensible default), and a piece of guidance phrasing that read as a global cap on Quarter’s whole turn when it was meant to constrain one specific response type: a genuine self-contradiction the model would have hit the moment two guided decisions were live in the same turn.
The part worth stealing
Before treating any A/B comparison as decisive, check that both arms actually got the same treatment, not “the same treatment as far as the experiment’s design intended,” but the same treatment as the logs on disk actually record. The gap between those two is exactly where this comparison’s real finding was hiding.
What Next
Shipping With a Watch, Not a Verdict
Five doubt cycles on one small registry guard, several defects introduced by the previous cycle's own fix, and the honest artefact that shipped with a WATCH instead of a declared-safe verdict.