Gemma 4 QAT Q4_0 Bench on Strix Halo
*UPDATED* These are Google's official Gemma 4 QAT Q4_0 GGUF models, served locally
through llama.cpp Vulkan/RADV on a Strix Halo APU.
QAT means **quantization-aware training**. Instead of taking a normal model and
quantizing it only after training, the model is trained or adapted while
accounting for the lower-precision format it will run in. The goal is to make a
small Q4 model keep more of the original model's behavior than a simple
post-training quantization.
## Host
**System:** AMD Ryzen AI Max+ 395 / Radeon 8060S, `gfx1151`
**Memory:** 128 GB unified LPDDR5X
**GTT ceiling:** 96 GiB class / large-GTT setup
**IOMMU:** enabled
**OS:** Linux Mint 22.3 / Ubuntu noble base
**Kernel:** `6.17.0-23-generic`
**Mesa / RADV:** Mesa `25.2.8` / RADV
**Backend:** llama.cpp Vulkan/RADV, Atomic llama.cpp TurboQuant fork for Gemma 4
assistant-head MTP
**ROCm:** installed, but these rows are Vulkan/RADV inference rows
## Models
**Main model:** `google/gemma-4-26B-A4B-it-qat-q4_0-gguf`
**Main model file:** `gemma-4-26B_q4_0-it.gguf`
**Main model size on disk:** `14,439,361,440` bytes / `13.45 GiB`
**Architecture:** Gemma 4 MoE, roughly 26B total / A4B-ish active lane
Other QAT models tested:
| Model | File size |
|---|---:|
| Gemma 4 12B QAT Q4_0 | `6,975,877,728` bytes / `6.50 GiB` |
| Gemma 4 26B-A4B QAT Q4_0 | `14,439,361,440` bytes / `13.45 GiB` |
| Gemma 4 31B QAT Q4_0 | `17,650,999,456` bytes / `16.44 GiB` |
## MTP Assistant Heads
The first QAT MTP probes borrowed the normal non-QAT Gemma 4 assistant heads.
Those loaded, but acceptance was weak. The better result came from using the
matching QAT assistant sources from Google and converting those assistant
checkpoints to Atomic/llama.cpp-compatible GGUF heads.
Official QAT assistant sources:
```text
google/gemma-4-12B-it-qat-q4_0-unquantized-assistant
google/gemma-4-26B-A4B-it-qat-q4_0-unquantized-assistant
google/gemma-4-31B-it-qat-q4_0-unquantized-assistant
```
Converted local assistant heads:
| Main model | QAT assistant head | Size |
|---|---|---:|
| Gemma 4 12B QAT | `gemma-4-12B-it-qat-assistant-MTP-Q8_0.gguf` | 444 MiB |
| Gemma 4 26B-A4B QAT | `gemma-4-26B-A4B-it-qat-assistant-MTP-Q8_0.gguf` | 441 MiB |
| Gemma 4 31B QAT | `gemma-4-31B-it-qat-assistant-MTP-Q8_0.gguf` | 491 MiB |
Conversion note: the assistant GGUF needs the `gemma4_assistant` metadata shape
that this Atomic llama.cpp build expects, including `n_embd_backbone` and target
architecture metadata. A public 31B QAT assistant GGUF I tried used different
metadata and did not load as-is. The 12B source repo uses a newer
`Gemma4UnifiedAssistantForCausalLM` config name, so I converted it through a
temporary config alias to the existing Gemma 4 assistant converter path. The
source weights were not hand-edited.
## Latest Measured Numbers
| Lane | Load to listening | Prefill | Decode | Normalized wall, 1150-in/2000-out | Two-slot aggregate | Notes |
|---|---:|---:|---:|---:|---:|---|
| Gemma 4 26B-A4B QAT Q4_0, plain F16 KV | ~4 s | 1194.4 tok/s | 59.4 tok/s | 34.6 s | 90.9 tok/s | best plain row |
| Gemma 4 26B-A4B QAT Q4_0, QAT MTP + Q8 KV | ~18 s | 729.3 tok/s | 71.4 tok/s | 29.6 s | 62.5 tok/s | best overall QAT lane |
| Gemma 4 12B QAT Q4_0, QAT MTP + Q8 KV | ~10 s | 539.9 tok/s | 45.6 tok/s | 46.0 s | 43.5 tok/s | strong small-model MTP lane |
| Gemma 4 12B QAT Q4_0, plain F16 KV | ~4 s | 666.5 tok/s | 25.7 tok/s | 79.5 s | 47.6 tok/s | plain baseline |
| Gemma 4 31B QAT Q4_0, QAT MTP + F16 KV | ~20 s | 203.6 tok/s | 19.1 tok/s | 110.4 s | 18.9 tok/s | works, but less efficient than 26B-A4B |
| Gemma 4 31B QAT Q4_0, plain Q8 KV | ~8 s | 204.2 tok/s | 11.0 tok/s | 187.4 s | 20.0 tok/s | best plain 31B row |
The main result: the 26B-A4B QAT model is the useful lane. Plain Vulkan already
gives about **59 tok/s decode** with very strong prefill, and the QAT-matched
MTP/Q8 path reaches about **71 tok/s single-stream** with much better acceptance
than the borrowed-head probe.
## Draft Acceptance
Current QAT-matched MTP rows:
| Model | MTP acceptance | Effective acceptance-adjusted decode |
|---|---:|---:|
| Gemma 4 12B QAT Q4_0 + QAT MTP head | 78.4% | 43.9 tok/s |
| Gemma 4 26B-A4B QAT Q4_0 + QAT MTP head | 91.8% | 71.4 tok/s |
| Gemma 4 31B QAT Q4_0 + QAT MTP head | 60.4% | 19.0 tok/s |
The 26B-A4B row is the standout. It keeps the fast decode lane and acceptance
is now high enough that I would treat it as the real QAT MTP result, not just a
speed probe.
31B is more of a tradeoff:
| 31B setting | Decode | MTP acceptance |
|---|---:|---:|
| `DRAFT_BLOCK_SIZE=3` | 19.1 tok/s | ~60% |
| `DRAFT_BLOCK_SIZE=2` | 16.5-17.1 tok/s | ~76% |
`DRAFT_BLOCK_SIZE=1` is not accepted by this build; the allowed range starts at
- `DRAFT_P_MIN` did not materially change the 31B acceptance in my short
sweep.
For comparison, the earlier borrowed-head QAT MTP rows were lower quality as
MTP stacks:
| Model | Borrowed-head acceptance | QAT-matched acceptance |
|---|---:|---:|
| Gemma 4 26B-A4B QAT Q4_0 + MTP | 56.9% | 91.8% |
| Gemma 4 31B QAT Q4_0 + MTP | 42.5% | 60.4% |
## Context Against Previous Local Gemma Rows
| Model / lane | Quant / path | Prefill | Decode |
|---|---|---:|---:|
| Gemma 4 26B-A4B non-QAT | UD-Q6_K_XL, plain Vulkan | 1002.8 tok/s | 44.8 tok/s |
| Gemma 4 26B-A4B QAT | Q4_0, plain Vulkan | 1194.4 tok/s | 59.4 tok/s |
| Gemma 4 26B-A4B QAT | Q4_0 + QAT MTP/Q8 KV | 729.3 tok/s | 71.4 tok/s |
| Gemma 4 31B non-QAT | Q6 plain Vulkan | 151.3 tok/s | ~8.1 tok/s |
| Gemma 4 31B QAT | Q4_0 plain Vulkan | 204.2 tok/s | 11.0 tok/s |
| Gemma 4 31B QAT | Q4_0 + QAT MTP/F16 KV | 203.6 tok/s | 19.1 tok/s |
| Gemma 4 12B QAT | Q4_0 plain Vulkan | 666.5 tok/s | 25.7 tok/s |
| Gemma 4 12B QAT | Q4_0 + QAT MTP/Q8 KV | 539.9 tok/s | 45.6 tok/s |
## Takeaway
On a 128 GB Strix Halo APU, Google's official Gemma 4 26B-A4B QAT Q4_0 GGUF is
a very strong local lane: about **59 tok/s plain** and about **71 tok/s** with
the QAT-matched MTP/Q8 setup.
The important update is that QAT-matched assistant heads matter. Borrowing the
normal non-QAT assistant heads was useful for proving that MTP could load, but
the matched QAT heads substantially improved acceptance, especially on the
26B-A4B row.
I have not verified these QAT MTP rows on stock upstream llama.cpp or vLLM
locally. The measured claim here is the Atomic llama.cpp TurboQuant fork on
Vulkan/RADV.
Posted Jun 6, 2026, 2:22 PM UTC