Pattern we use to make LLM outputs more reliable in production

We kept running into the same issue with LLMs in prod: answers sounded great… until they weren’t.

Everything looked confident and polished, but sometimes key context was missing or stale, and things broke. Prompt tweaks helped a bit, but it felt like patching symptoms.

So we tried a multi-agent setup:

• Primary agent → generates the answer
• Context agent → checks it against real system state
• Skeptic agent → tries to break the logic
• Arbiter → decides if it ships

The skeptic ended up being the MVP. Its only job is to find flaws. When it can’t break the answer, our error rates drop significantly.

Big takeaways:

• Most hallucinations come from missing info, not bad reasoning.
• Agents need different incentives, not just different prompts.
• Latency increases, but trust improves a lot.

Not magic, but it’s the most reliable pattern we’ve found for predictable LLM behavior in production.

Curious what others are doing for validation and trust?