Anyone tried consensus loop engineering ?

context: small team, dogfooding on our own repos, no external users, so grain of salt.

for a while our setup was a few agents reviewing each change and only merging if they agreed. the idea was fine. the problem was the reviewers themselves were inconsistent. same PR, run it monday it passes, run it wednesday one of them suddenly blocks on something it waved through before. the non-determinism wasn't in the code, it was in the judge.

we tried fixing it the obvious way, bigger review prompts, more rules stuffed inline. that got worse honestly, under a long context the rules just quietly fell off and the reviewers drifted back to vibes.

what actually worked was dumb. we pulled the review out of the prompt and wrote it down as a skill the agents load every single run. independent passes that can't see each other's verdict, and any single "not sure" stops the merge instead of getting averaged into a yes. same rules, every time, versioned in the repo next to the code.

once the judgment was a fixed artifact instead of something we re-explained each run, the consensus stopped wobbling. it's slower and occasionally pedantic, i watched it stall a perfectly clean change for a couple extra rounds last week. but it stopped passing things it would've blocked on a different day, which was the whole point.

how are you all keeping review or eval criteria consistent across runs? do you put it in a skill or a file, fine-tune it in, or just trust the prompt to hold?