What are your configuration tricks to increase prefill and decode speed?
I'm not an expert on this but this is what I do. I have an old macbook with 32gb of unified ram so my definition of usable is 10 to 15 t/s. I know some of you want more than 50 t/s+ but many of us are not in that territory yet :)
-
Play around with quants. Although we have to know we are using a dumber version of the model, so be careful.
-
Trying MTP, MLX (if mac), Unsloth models, and other speculative decoding methods. Can't wait to try DSpark when it's available btw.
-
Disabling reasoning.
-
Using KV cache quantization (set to Q8).
-
Not setting up a big context if not needed.
What else are you doing? I read you! thanks.
Posted Jul 5, 2026, 8:54 PM UTC
22