We've been researching evaluation platforms while scoping our rollout. Ultimately the team came to these specific criteria to evaluate the different platforms. Curious what other orgs are including in their evaluation axes (pretty sure there are no standards yet) — we're set on buying rather than building, so please don't recommend we build our own solution.

The axes that actually mattered to us:

• Tracing/observability depth — span-level detail, latency/cost breakdowns, session-level views vs. just prompt-level
• Evaluation methodology — built-in metric libraries vs. bring-your-own, LLM-as-judge support, human-in-the-loop annotation
• CI/CD integration — can this gate a deploy, or is it purely post-hoc
• Agent-specific evaluation — tool-call correctness, trajectory/step evaluation, not just final-output scoring
• Red-teaming / adversarial testing — built in or bolted on
• Governance — versioning, approvals, audit trail, RBAC
• Framework lock-in — is it framework-agnostic
• Deployment model — SaaS-only vs. self-hosted/open-source option (we need self-hosting since we're a large org)

Feature comparison (roughly, as of our research):

(CI/CD gate support and framework lock-in mattered to us too, but cutting them kept the table from turning into a spreadsheet — happy to share those details in comments if useful.)

A few things that stood out once we laid it out this way rather than as prose:

"Agent evaluation" means different things depending on the vendor. Some platforms score the final output of a multi-step run; others actually evaluate the trajectory — did it call the right tools, in a reasonable order, without looping or cheating. If you're running agentic workflows rather than single-turn Q&A, this distinction matters more than any other row in the table.

Governance and evaluation are converging, but unevenly. ConfidentAI and Galileo build governance in; LangSmith treats it as secondary; the OSS libraries (DeepEval, RAGAS, Promptfoo) don't really have a governance story at all — you're expected to build that layer yourself or pair with something like a prompt-versioning tool.

Self-hosting is a real fork in the road, not a checkbox. If your org has data residency or compliance requirements, that alone probably eliminates half this list before you even get to feature comparison.

Red-teaming is still spottier than the marketing suggests. ConfidentAI, Galileo, and Promptfoo have genuine red-teaming capability. Most others treat it as a roadmap item or expect you to bring your own adversarial test set.

One thing this table can't capture well: actual day-to-day usability and how these hold up once you're running hundreds of evaluations a day instead of dozens. If anyone has run more than one of these side by side at volume, curious where reality diverged from the pitch.