I Benchmarked the Same Model on Two 'SWE-bench' Variants. Scores Were 20 Points Apart.

Headlines say GPT-5.4 scores 57.7% on SWE-bench. GLM-5 scores 77.8%. So GLM-5 wins, right?

No. They sat for different exams.

SWE-bench Pro is roughly 11k GitHub issues, hand-curated by the SWE-bench team and edited for clarity. Medium difficulty. If you care about SWE evals, it's the one to look at.

SWE-bench Verified is a much tighter set: about 300 issues from the original SWE-bench, picked over by multiple contributors until everyone agreed the labels were right. The problems are harder and the pool is much smaller, which means you get a cleaner read on the examples you have at the cost of seeing less of the real distribution.

GPT-5.4's score is on Pro. GLM-5's is on Verified. That 20-point gap isn't a quality difference but one in which test each one took.

this isn't specific to SWE-bench either. MMLU vs MMLU-Pro. ImageNet vs ImageNet-A. The names look like variants of one number. They aren't. Someone posts "SWE-bench" in a thread and you assume you know what was measured. Usually you don't.

The fix is to run both models on the same benchmark.

Stratix makes this less of a chore (run any model against any benchmark on your own data). But product pitch aside: if you haven't run the models on the same tasks, what you have is a comparison of benchmarks dressed up as a comparison of models.