Evaluating LangChain agents beyond final output

I’ve been running a lot of experiments with agents built on LangChain recently. Getting them to work wasn’t the hardest part. Getting them to behave consistently is.

Once you combine:

• tool calling
• retries
• multi-step reasoning
• branching logic
• memory/state

the system becomes less “a prompt” and more “a distributed workflow”.

And evaluating that workflow is surprisingly tricky.

Two runs with the same input can:

• take different tool paths
• retry at different steps
• recover from errors differently
• reach the same final answer via completely different trajectories

If the final answer is correct, is that enough? Or should we care about how it got there?

What I’ve noticed is that many failures aren’t LLM failures.

They’re orchestration failures.

• retry policies that amplify small errors
• tool outputs that slightly mismatch expected schemas
• state drifting over multiple steps
• subtle branching differences that compound

From the outside, the agent “works”. Internally, it’s unstable.

I’ve started treating agent evaluation more like system observability:

• snapshotting full execution traces
• comparing repeated runs
• looking at divergence points
• tracking stability across multiple executions

Not just “did it answer correctly?” But “does it behave consistently under repetition?”

For those building with LangChain (or LangGraph):

• Are you evaluating trajectories, or just outputs?
• Do you test multi-run stability?
• How do you detect silent orchestration failures?
• Are you using built-in tracing only, or something beyond that?

Curious how others here are thinking about reliability at the workflow level.