LM Studio - Spec Decoding - No improvement gains no matter which configuration
I'm not sure what I've been doing wrong as I've tried a large combination of different models + spec decoding in hopes for a performance increase for tps but to no avail in LM Studio after spending more than 6-7 hours frustratingly debugging this.
I've tried:
Deepseek R1 Distill Qwen 32b + Deepseek R1 Distill Qwen 1.5b
Deepseek R1 Distill Qwen 7b + Deepseek R1 Distill Qwen 1.5b
Qwen 2.5 32b Coder + Qwen 2.5 1.5b
Mistral Small 3.1 24b + Mistral Small 3.1 0.5b Draft
QwQ 32b + Qwen 2.5 1.5b
QwQ 32b + QwQ 0.5b
I've tried combinations between q6 to q4 as well. All draft models were running above 90 tps to 140 tps while the main model was between 15 to 30 tps.
No performance gains in all of these combinations and actually a massive performance loss ranging from 1 token down to 15 tps loss despite > 35 - 40% of the draft tokens being accepted. One of the most possible reason for this loss is because despite the draft model being loaded into the GPU (in the case of Deepseek 7b + Deepseek 1.5b, the draft model )
PC specs: 1x RTX 3090, 1x RTX 4070 Laptop, i9 14900HX running Windows 11
Am I configuring something incorrectly in LM Studio?
I'm currently considering on switching to Tensor LLM for higher performance and tps compared to llama.cpp with a OpenAPI wrapper for TensorLLM to serve on OpenWebUI. Any thoughts on this matter and if anyone has any experience with Tensor LLM on Windows, your experience is much appreciated!
Posted Apr 8, 2025, 9:06 PM UTC