How are you evaluating multi-step reliability before deploying LangChain agents?

One thing that keeps bothering me with agent workflows is that a single successful run does not necessarily mean the change is safe to ship.

With tool calling, retries, branching, and state, the final answer can look okay while the workflow underneath becomes less stable. We started replaying saved real cases before deploy and repeating the same runs on purpose, and that was where some cases started to look flaky instead of consistently healthy.

That made me realize that “looks fine” in a few spot checks is not the same as “safe to deploy.”

So I’m curious how people here handle this in practice:

• Do you evaluate only the final output, or workflow stability too?
• Do you repeat runs on the same saved cases to catch flaky behavior?
• What would actually make you stop a release before shipping?

Especially interested in teams changing prompts, models, or agent workflow logic regularly.