Qwen3.5 prefill latency extremely slow with large context

Here is a simple human sounding Reddit draft written naturally without formatting or symbols:

Hi everyone, I wanted to check if others are seeing similar behavior with Qwen3 and Qwen3.5 models on llama.cpp or if there is any known workaround.

I am running llama server built from the latest commit as of today on my AMD R9700 32G + 64GB system RAM.

llama server version 8180 commit d979f2b176217a062c57c24355db8c1dc3dfc7d5
built with GNU 13.3.0 on Linux x86_64
GPU AMD Radeon Graphics gfx1201 using ROCm

I noticed that Qwen3.5 models have much higher prefill latency compared to Qwen3 models even when model size and quantization are very similar.

For example when using small input prompts the performance looks normal.

Small input example
prompt tokens 28
prompt time about 57 ms
prompt per second around 491 tokens per second
generation speed about 63 tokens per second
first token appears almost instantly

But when sending large input content such as long articles or extracted PDF text the first token takes a very long time even though generation speed after that stays fast.

Large input example
prompt tokens about 4907
prompt time about 90190 ms
prompt per second around 54 tokens per second
generation speed still around 62 tokens per second

So decode speed looks fine but prefill latency becomes extremely slow with large context.

I also noticed that Qwen3 32B Q4 and Qwen3VL 32B Q4 process the same large input much faster while Qwen3.5 27B Q4 and Qwen3.5 35B A3B MXFP4 are significantly slower during prompt processing.

Has anyone else observed slower prefill performance specifically with Qwen3.5 models on llama.cpp especially on AMD GPUs

Is this expected due to architecture differences or are there flags build options or runtime settings that improve prefill speed

Any advice or shared experience would really help since decode performance itself looks good but time to first token becomes very high for real workloads like RAG or long documents.

Thanks in advance.