Your retriever can return the correct document and your RAG system can still hallucinate. Your agent can call the right tool and your production system can still corrupt state.

When an LLM-backed chat feature fails, the user sees a bad answer. When an autonomous agent fails, it loops on a bad tool call, writes a malformed payload to a downstream system, or burns through a token budget with nothing to show for it. Standard LLM monitoring the kind built for single prompt-response pairs was never designed to catch that.

# What you'll learn



* Why static benchmark scores don't predict agent reliability in production
* The difference between LLM tracing and agent observability
* A 3-tier evaluation framework: step-level, trajectory, and output
* How to combine Pydantic schema validation with OpenTelemetry tracing
* A production readiness checklist you can run against your own agent today



# 1. The non-deterministic agent gap

Foundation models score well on static benchmarks like GPQA or HumanEval. Those scores describe reasoning capacity on a fixed test set. They don't describe what happens when that same model runs inside a plan → execute → verify → re-plan loop against live systems.

That gap matters because the two failure modes look nothing alike. A benchmark failure is a wrong answer. A production agent failure is usually not a reasoning failure at all it's system integration state drift: a tool call with the wrong argument shape, an API that returns a shape the agent wasn't expecting, or a retry that repeats an action that already succeeded.

If you haven't already bounded how many steps an agent can take before it has to stop, start there see AI agent loops, stopping conditions and guardrails. Everything below assumes that ceiling already exists.

Here's the failure path end to end: a user trigger reaches the agent core, the agent selects a tool call, the tool execution engine validates and runs it against an external API, and every step along the way emits a trace span to an OpenTelemetry collector. When the external API rejects the call an HTTP 422 here that error has to come back to the tool execution engine as structured telemetry, not just a stack trace in a log file.



Architecture diagram: production agent observability

https://preview.redd.it/aabo22ddd8ph1.png?width=1800&format=png&auto=webp&s=d7026745141689a2414ba5f2a82ec336b9b6736d

# 2. LLM tracing vs. agent observability

Basic LLM tracing checks prompt-to-response performance: one request, one response, one span. Agent observability has to trace execution paths across multi-turn state graphs, because the unit of work isn't a single completion it's a chain of tool calls that share state.

The differences that matter in practice:



* Primary unit: a single completion span (basic tracing) versus a full execution graph — root trace plus child spans (agent observability)
* Monitored payload: input text, output text, and token count (basic) versus tool schemas, intermediate state, and raw HTTP responses (agent)
* Failure telemetry: high latency, HTTP 500s, and thumbs-down feedback (basic) versus state corruption, schema validation errors, and loop cycles (agent)
* Evaluation timing: post-hoc offline batch evaluation (basic) versus real-time runtime assertions and step guardrails (agent)



If your current setup only captures the first column, you have LLM tracing, not agent observability. They're not the same instrumentation with more dashboards they answer different questions.

# 3. The 3-tier production agent eval framework

Reliability isn't one measurement. It's three, and they build on each other:

3-tier evaluation stack

https://preview.redd.it/45sqt02hd8ph1.png?width=1800&format=png&auto=webp&s=26ae62ebbfabefecff4f463d4eb3a996b9960404

* Step-level evaluation: does each individual tool call conform to its schema contract? This is the cheapest check to run and it should run synchronously, before the call executes.
* Trajectory evaluation: across the whole run, is the agent taking an efficient path, or is it re-calling the same tool with near-identical arguments a loop pattern? See agent tool error cascade for a worked example of what an uncaught step-level failure does to the trajectory above it.
* Output evaluation: is the final answer actually grounded in what the tools returned, and does it match the business requirement not just "is it fluent"?



A trajectory can look efficient and still produce a wrong output. An output can look correct and still have gotten there through three retried, near-identical tool calls that quietly tripled your cost. You need all three tiers, and you need step-level passing before trajectory-level telemetry is worth trusting.

For the broader vocabulary faithfulness, groundedness, LLM-as-judge, regression suites see LLM evaluation fundamentals for production. This article is the agent-specific layer on top of that foundation.

# 4. Implementation: schema validation and OpenTelemetry tracing

The pattern below combines Pydantic schema validation with OpenTelemetry tracing so that a malformed tool call from the model produces a structured, queryable failure instead of an unhandled exception three layers downstream.

python

    import json
    from typing import Dict, Any
    from pydantic import BaseModel, Field, ValidationError
    from opentelemetry import trace
    from opentelemetry.sdk.trace import TracerProvider
    from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter
    
    # Initialize OpenTelemetry once, at process startup
    provider = TracerProvider()
    provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
    trace.set_tracer_provider(provider)
    tracer = trace.get_tracer("agent.observability", "1.0.0")
    
    
    class DatabaseQuerySchema(BaseModel):
        """Contract for a database_query tool call. This is the shape the
        agent's tool call arguments must satisfy before we ever touch
        application state."""
        table_name: str = Field(description="Target database table")
        limit: int = Field(default=10, ge=1, le=100, description="Row limit")
        filters: Dict[str, Any] = Field(default_factory=dict)
    
    
    def execute_agent_tool(raw_tool_args: str) -> Dict[str, Any]:
        with tracer.start_as_current_span("agent_tool_execution") as span:
            span.set_attribute("agent.tool.name", "database_query")
            span.set_attribute("agent.tool.raw_payload", raw_tool_args)
    
            # Step-level eval: schema validation happens before execution,
            # not after — this is the cheapest place to catch a bad call.
            try:
                parsed = json.loads(raw_tool_args)
                validated = DatabaseQuerySchema(**parsed)
    
                span.set_attribute("agent.tool.validated", True)
                span.set_attribute("agent.tool.table", validated.table_name)
                span.set_attribute("agent.tool.limit", validated.limit)
    
                # Real execution goes here — mocked for illustration.
                result = {"status": "success", "rows_returned": 2}
                span.set_attribute("agent.tool.status_code", 200)
                return result
    
            except (json.JSONDecodeError, ValidationError) as exc:
                # Trajectory guardrail: log the precise schema failure so a
                # downstream re-plan step has something structured to act on,
                # instead of a raw string it has to re-parse.
                span.set_attribute("agent.tool.validated", False)
                span.record_exception(exc)
                span.set_status(trace.Status(trace.StatusCode.ERROR, str(exc)))
                return {
                    "status": "error",
                    "error_type": "SchemaValidationError",
                    "message": f"Tool call failed schema validation: {exc}",
                }
    
    
    if __name__ == "__main__":
        # limit=500 violates the le=100 constraint — this is what a
        # malformed LLM tool call actually looks like in practice.
        bad_payload = '{"table_name": "users", "limit": 500}'
        print(json.dumps(execute_agent_tool(bad_payload), indent=2))

The point of this pattern isn't the code itself it's that the schema failure becomes a span attribute (agent.tool.validated: false) instead of an exception that only shows up in a log aggregator hours later. That's what makes step-level evaluation queryable at the trajectory level: you can ask "how often did tier 1 fail this week" as a trace query, not a grep.

# 5. Production readiness checklist



* Instrument native tracing: export OpenTelemetry-compliant spans for every model call, vector search, and tool execution
* Enforce schema contracts: wrap every tool execution interface in strict Pydantic (or Zod, for TypeScript agents) schemas
* Set hard trajectory caps: limit execution to a maximum step count (max\_turns) to prevent infinite loops see agent loops, stopping conditions and guardrails
* Implement step assertions: validate HTTP status codes and payload schemas before passing external API output back into model context
* Run trajectory regression suites: maintain a library of historical execution traces and replay them against new model or prompt versions to catch drift



# 6. Frequently asked questions

Why can't I just use standard LLM monitoring platforms? Standard LLM monitoring tracks individual API requests and responses. Agents execute non-linear graphs across multiple turns, state stores, and APIs. Pinpointing which exact tool execution or state update caused a downstream failure requires tracing the full trajectory, not a single span.

Should I use open-source or commercial observability tools? Start with an open, vendor-neutral standard OpenTelemetry regardless of which platform you view traces in. Tools like Langfuse, Arize Phoenix, or SigNoz can all consume standard OTel traces, which means you're not locked into one vendor's SDK if you switch tools later.

Do I need all three evaluation tiers from day one? No. Step-level schema validation is the highest-leverage tier to build first it's cheap, synchronous, and catches the most common failure mode (malformed tool arguments) before it ever touches application state. Add trajectory and output evaluation as your agent takes on longer, higher-stakes tasks.

# When not to use this framework

If your agent makes a single tool call per run with no multi-step loop, full trajectory evaluation is probably overhead you don't need yet step-level schema validation and basic output checks will cover you. Reach for the full 3-tier framework once an agent can take more than one action per task, because that's the point where trajectory-level failures (loops, redundant calls) become possible.

# Related learning



* Building Effective Agents - Covers AI agent loops, execution bounds, and setting deterministic stopping conditions. ([https://www.anthropic.com/research/building-effective-agents](https://www.anthropic.com/research/building-effective-agents)) 
* Demystifying LLM Evals- Outlines practical evaluation fundamentals for testing and deploying LLM applications in production. ([https://www.anthropic.com/engineering/demystifying-llm-evals](https://www.anthropic.com/engineering/demystifying-llm-evals))
* Introducing the Model Context Protocol - Explains the open standard for securely connecting AI agents to tools and internal APIs via MCP servers. ([https://www.anthropic.com/news/model-context-protocol](https://www.anthropic.com/news/model-context-protocol)) 
* Building Effective Agents- Details architectural design patterns including Workflows, Routing, Parallelization, and Orchestrators. ([https://www.anthropic.com/research/building-effective-agents](https://www.anthropic.com/research/building-effective-agents))
* How We Built Our Multi-Agent Research System - Addresses handling tool execution failures, state management, and mitigating error cascades.([https://www.anthropic.com/engineering/multi-agent-research-system](https://www.anthropic.com/engineering/multi-agent-research-system))



AI agent observability: why evaluation matters more than the model