Why do the Qwen 3.5 series benchmark better than Qwen 3 series?

Question

As we all know, Qwen 3.5 is on a tear. It scores very well on benchmarks (cf https://pastes.io/benchmark-60138 for small model comparison).

I'm curious: how much of this is "think harder" being baked in (even with settings turning off thinking mode, the model appears to consume thinking tokens, judged by wall clock time) versus genuine architectural improvement?

At first blush, the dramatic boost on HMMT25 (math) suggests "think harder" is the secret sauce. But then GPQA Diamond is factual knowledge and reasoning, and that's also massively improved.

Has anyone actually benchmarked Qwen3.5-4B with thinking disabled?

Because if the architectural changes alone account for most of the gain, that's interesting.

If thinking tokens are doing 80% of the work, that's also interesting, just in a different direction.

What's your read re: the 11 secret herbs and spices?

Posted Mar 6, 2026, 4:11 AM UTC

39
Loading comments…