r/LocalLLaMA•
regunakyle

PSA: You may not need to quantize spec draft when using MTP

Discussion

Using `--spec-draft-type-k q4_0 --spec-draft-type-v q4_0` might actually decrease your context size!

With quantized spec draft, my context size is 83200. Without it (i.e. using the default of fp16 spec draft), context size increased to 91648.

I reported this in a llama.cpp discussion and am17an (the GOAT behind MTP in llama.cpp) confirmed my findings as expected:

https://github.com/ggml-org/llama.cpp/discussions/24102

Posted Jun 5, 2026, 4:41 AM UTC

2038
Loading comments…