Qwen 3.6 35b a3b dropping nearly 50% t/s with only slight VRAM:Shared RAM offload

I've been trying to fine-tune Qwen 3.6 35b a3b for speed and context size, and I've noticed that it can load either in what I'm calling "fast mode" (~1700t/s prefill, ~60t/s decode, on initial test prompt), or "slow mode" (~500t/s prefill, ~35t/s decode). I think it's related to slight differences in available VRAM from other system use - it just seems extremely sensitive to even slight (200MB) changes that tip it over the edge, from "fast mode" to "slow mode". It surprises me a little because the model is MoE, and I would've thought if VRAM overspills a bit more into shared RAM, it would start with light-usage experts/layers, barely impacting model speed. Is it because the GPU driver or Vulkan starts to handle this automatically once the model is loaded, with no knowledge/care for the model weights moved?

I can trigger "slow mode" either from model load by just ramping up context size, or by starting in "fast mode", then using some VRAM (e.g. YouTube videos) to force some VRAM to get shifted to shared RAM.

I would like to ask for help, if anyone knows what is happening in this situation, or knows ways to avoid the problem? Is there anything I can improve in llama.cpp params (pasted below) to "protect" certain model weights from getting offloaded?

A strategy I've found is to set `fit-target` quite large (e.g. 2GB) to allow llama.cpp to "smartly" distribute weights at load-time, so that if VRAM usage increases a small amount during runtime, there's enough buffer that (I guess) Vulkan or GPU driver doesn't shift random model/cache stuff into shared RAM and kill model speed. Is that about my best option, if there's no way to "protect" llama's initial model-load from arbitrary VRAM shifts later?

My setup is llama.cpp (Vulkan backend), Windows 11, RX 9070 XT (16GB VRAM), 32GB RAM. Due to AMD Shared Access, the 9070 XT also can (and does) use up to 16GB of system RAM directly. I'm using unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL.

[Qwen3.6 35B MTP]
hf = unsloth/Qwen3.6-35B-A3B-MTP-GGUF:UD-Q4_K_XL
fit-target = 512
cache-type-k = q5_1
cache-type-v = q5_1
c = 120000
spec-type = draft-mtp
spec-draft-n-max = 2
cache-type-k-draft = q4_0
cache-type-v-draft = q4_0
reasoning-budget = 4096
chat-template-file = froggeric_chat_template_213.jinja
flash-attn = true
load-mode = none
reasoning = on
reasoning-preserve = true
chat-template-kwargs = {"preserve_thinking": true, "enable_thinking": true}
temp = 0.6
top-p = 0.95
top-k = 20
min-p = 0.0
repeat-penalty = 1.0
presence-penalty = 0.0

offline = true
metrics = true
np = 1
jinja = true
device = Vulkan0
threads = 8
batch-size = 4096
ubatch-size = 1024