Running Gemma 4 QAT 12B on an 8GB GPU at 16k context — measured the KV-cache tradeoffs

QAT (quantization-aware training) is the headline of the Gemma 4 release, so I ran it on actual old hardware and measured three things: quality, speed, and whether the 12B fits an 8GB card at 16k.

Quality (Unsloth's top-1 agreement vs the full model, UD-Q4_K_XL vs naive Q4_0):

• 12B: 88.76% vs 74.08% · E2B 98.16 vs 89.29 · 31B 96.67 vs 87.91

So naive 4-bit drops a lot; QAT+dynamic keeps most of it, at ~72% less memory than BF16. Honest note: that gap is vs naive Q4_0 — vs a good Q4_K_M it's much smaller (on my own coarse probes I couldn't separate QAT from a solid Q4_K_M). Also: stick to UD-Q4_K_XL — Unsloth notes higher precision degrades these QAT models.

Speed/size on a single GTX 1080 Ti (num_ctx 8192, 100% GPU):

• regular Q4: 28.3 tok/s / 7.6GB · Google QAT: 31.0 / 7.5GB · Unsloth QAT: 30.8 / 7.2GB → QAT ~9% faster, slightly smaller. Modest, but a 12B at ~30 tok/s on an 8-year-old card is the real story.

Fitting the 12B on 8GB at 16k (the question I actually get asked). Weights ~7GB → ~1GB for KV. Measured footprint at 16k, single GPU, flash-attn on:

Default f16 KV won't fit once you count driver/display reserve — quantize KV to q8_0 → 7.4GB, 100% GPU, negligible quality cost. Neat detail: q8 and q4 KV differ by only ~0.2GB at 16k, because Gemma's sliding-window attention keeps the KV small — the footprint is dominated by the weights, and KV quant just buys the last bit you need to slip under 8GB.

Recipe: OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 + num_ctx 16384 + all layers on GPU (verify ollama ps shows 100% GPU).

Full writeup with the caveats: https://bric.pe.kr/blog/gemma-4-qat-1080ti-8gb-12b-16k-measured

Anyone running the 12B QAT on 8GB cards — what KV type / context are you landing on?