APEX MoE quantized models boost with 33% faster inference and TurboQuant (14% of speedup in prompt processing)
I've just released APEX (Adaptive Precision for EXpert Models): a novel MoE quantization technique that outperforms Unsloth Dynamic 2.0 on accuracy while being 2x smaller for MoE architectures.
Benchmarked on Qwen3.5-35B-A3B, but the method applies to any MoE model. Half the size of Q8. Perplexity comparable to F16.
Works with stock llama.cpp with no patches. Open source (of course!), with <3 from the github.com/mudler/LocalAI team!
Perplexity by itself doesn't say the full story. KL divergence tells a story perplexity doesn't:
Tiers for every GPU:
- I-Quality: 21.3 GB -- best accuracy
- I-Balanced: 23.6 GB -- best all-rounder
- I-Compact: 16.1 GB -- fits 24GB GPUs
- Mini: 12.2 GB -- fits 16GB VRAM
With TurboQuant, at 8K context, every APEX tier gets ~14% faster prompt processing (this is being benchmarked with a DGX Spark):
Models: http://huggingface.co/mudler/Qwen3.5-35B-A3B-APEX-GGUF
Method + technical paper: http://github.com/mudler/apex-quant
Run locally: http://github.com/mudler/LocalAI
Original post on twitter/X: https://x.com/mudler_it/status/2039364812463853708
Posted Apr 1, 2026, 8:13 PM UTC