How I'm testing AI agent improvements without losing my mind

So I built an AI agent that works... but it's obviously not perfect. And now the real work begins - constant improvements.

The problem: How do I know if my "improvement" is actually better than what's running in production?

This is what's been blocking me from confidently improving the agent. I could spend days tweaking prompts, but still not feel confident pushing to production.

Why I didn't use n8n's Evaluation feature:

It felt like overkill for my use case:

• Too complex to implement
• Requires automated scoring that doesn't fit my scenario
• My agent can give many "good" answers OR fail in many different ways

I believe at this stage, I need to manually review the responses myself.

Yes, I could build another agent to auto-evaluate responses, but that would pull me away from the actual focus right now.

My current solution:

For each improvement I make, I create a new test agent with that change embedded.

After testing it in a separate workflow and seeing it works well, I move it to production - but outside the main workflow that real users interact with.

Here's how it works in practice:

I have 5 test agents running in the background for every user question. I collect all their responses into a Google Sheet and review them manually.

I can see where each agent needs improvement and iterate in production without affecting users.

Is there a more efficient way to do this?

I'd love to hear how others are handling this. What's your workflow for testing AI agent improvements?

( I wanted to share here an image of the workflow but couldn't find how....)