How do you run qwen3 next without llama.cpp and without 48+ gig vram?
I have a 96g and a 128g system, both are ddr5 and should be adequate for 3b active params. I usually run moe like qwen3 30b a3b or gpt oss 20b / 120b with the moe layers in cpu and the rest in rtx 3080 10gb vram.
No GGUF support for qwen3 next so llama.cpp is out. I tried installing vllm and learned it cannot use 10g vram and 35g from system ram together like am used to with llama.cpp. I tried building vllm from source since it only has gpu prebuilds and main seems to be broken or to not support unsloth bitsandbytes (https://huggingface.co/unsloth/Qwen3-Next-80B-A3B-Instruct-bnb-4bit) Has anyone had success running it without the entire model in vram? If so, what did you use to run it, and if it is vllm, was it a commit from around sept9 ~ 4 days ago that you can provide the hash for?
Posted Sep 13, 2025, 10:47 PM UTC