What My Project Does

LLMAssert is a Python assertion library for verifying LLM output. It registers as a pytest plugin, installs via pip with optional extras (pip install "llm-assert[anthropic,semantic]"), and integrates directly into existing Python test suites.

result = v.assert_that(prompt).is_valid_json().semantic_intent_matches("a summary").run()
assert result.passed

Semantic checks run locally on a 22MB sentence-transformers model with no API calls for scoring. Structural checks (JSON, schema, regex, key presence) are deterministic and cost nothing. Behavioral checks run the model N times and use Wilson score confidence intervals so you know if a pass rate is real or just luck.

Drift detection catches silent model updates. Tested across real GPT-4o version pairs and Anthropic model migrations (Haiku to Sonnet). Score standard deviation across 100 identical runs: 0.0, which is expected since cosine similarity on the same embedding model with the same input is deterministic by definition.

• 458 tests total, 450 pass, 8 skip (need live API keys not present in CI)
• No accounts, no dashboards, no hosted anything
• Zero telemetry, verified by intercepting every outbound HTTP request during a full suite run

Target Audience

Production use. Engineers running LLM-powered features who need CI-compatible assertions that don't add nondeterminism to the test layer. Also useful for anyone evaluating model migrations or version upgrades who needs to quantify behavioral drift before and after.

Comparison

Most alternatives (DeepEval, Promptfoo, LangSmith, Braintrust) use LLM-as-judge for semantic scoring, which means the evaluation layer itself introduces variance. If the same input can produce a different score depending on when you run it, that's a problem for CI gates and regression tracking. LLMAssert uses local embeddings instead, so scores are deterministic, reproducible, and don't cost anything per assertion. LLM-as-judge can catch subtler things like sarcasm or tone, but for the core job of pass/fail assertions in a pipeline, stable scores matter more than deep interpretation.

In a measured 3-case suite run, LLMAssert made 3 provider API calls and 0 telemetry calls. The benchmarks are intentionally small because they're meant to demonstrate methodology and reproducibility, not generalize across all workloads. The scripts and raw JSON results are in the repo so anyone can run them against their own suite. Native drift detection across model versions is also built in, which at the time of writing I haven't found in the alternatives listed above.

Open source under Apache 2.0: https://github.com/moonrunnerkc/llm-assert