Distil: context compression for agents, gated by a statistical non-inferiority test — compressed context matched full context on 500 SWE-bench tasks

Every context compressor I've tried asks you to trust that squeezing the context won't change what the agent does. I wanted a compressor that proves it instead.

Distil's bet: for an agent, decision-equivalence (same tool calls, same output) matters, not byte-equivalence. So it gates each compression operating point through a paired non-inferiority test (TOST/McNemar) on a corpus of real trajectories — anything that would change the agent's decisions is rejected before it ships.

• Stdlib-only, zero runtime deps — runs anywhere, imports on stock Python 3.9+
• Reversible tier is byte-recoverable; the agent can expand any digest on demand
• Works as a proxy (point BASE_URL at it) for Anthropic / OpenAI / Gemini shapes

Result on 500 SWE-bench Verified (official harness): 42.0% vs 39.2% for full context — a certified tie (CI includes 0), i.e. compression that provably didn't hurt. Aggressive compression does break decision-equivalence and I don't hide that.

The gate exists because my first version failed. It summarized old turns aggressively — best savings on paper — and end-to-end task success collapsed when I ran it through the real harness: every individual turn looked reasonable, but the agent made different decisions and failed tasks it used to solve. Same wreck taught me the caching lesson: rewrite anything near the top of the conversation and you invalidate the prefix cache (API prompt caching, or your KV/prompt cache if you're serving locally), so the "compressed" run can cost more than the full one. Hence the two design rules: prefix stays byte-stable, tail gets compressed, and nothing ships without proving decision-equivalence on real trajectories. Per-turn token counts flatter every compressor; the harness doesn't.

10-second demo, no API key:

uvx --from distil-llm distil bench

Repo: https://github.com/dshakes/distil — feedback on the methodology very welcome.