I compared Qwen3.6-35B-A3B (with MoE expansion) vs Qwen3.8-27B dense at 32K and the dense model lose on GPQA-Diamond — not convinced by my own results, need help

Hi all,

I'm the author of the moe-expansion branch in llama.cpp (vagrillo). I ran a comparison on GPQA-Diamond (all 198 questions) between:

• Qwen3.6-35B-A3B, Q8_0, expanded routing → 84.34% (167/198)
• Qwen3.8-27B dense, Q8_0 → 83.84% (166/198)
• Qwen3.6-35B-A3B, Q8_0, native routing → 81.82% (162/198)

Greedy, 32K budget, same prompt, one RTX 5000 PRO 48GB, one run per config.

Why I'm not convinced:

• 35B expanded vs 27B dense is 1 question apart — that's parity, not a win.
• Expansion vs native on the 35B is 12 wins / 7 losses (net +5), but not significant at n=198.
• 10 of the 12 expansion wins are Chemistry only; Physics is saturated, Biology flat.
• One run, one seed.

What I'd like help with:

1. What other benchmarks pair well with GPQA-Diamond for this kind of routing comparison ?
2. How many runs / what setup would you consider enough for a paired comparison?
3. Is a Chemistry-only effect plausible, or a red flag?
4. Anyone else seeing a real MoE expansion effect on Qwen MoE models?

Thanks — I'm not claiming this is a result (the difference it's only 1 answer), I'm trying to understand my own numbers

the comparison results: https://htmlpreview.github.io/?https://github.com/vagrillo/llama.cpp/blob/moe-expansion/benchmark/RUN1209/finalcompariso1209.html