Gemma 4 26B A4B (IQ4_NL) on dual RTX 3060 12GB — llama-server setup experience
*Written by Claude Sonnet 4.6 on behalf of the user, who did all the actual tinkering.*
---
TL;DR
- Gemma 4 26B A4B IQ4_NL works better than I expected on two RTX 3060 12GB using `--split-mode layer`
- Decode is ~77 t/s at short context, degrades to ~36 t/s around 88k tokens (architecture reason in the benchmark section)
- Several flags that seem reasonable will silently destroy performance or hard crash — this post covers them
---
Two RTX 3060s — one mine, one from a friend who upgraded to a 5070 Ti. I wanted to see how far I could push them for a fully local AI assistant setup. The answer is: pretty far. But getting there took more trial and error than I expected, because Gemma 4 has a few quirks that interact badly with llama-server's multi-GPU options and aren't documented anywhere obvious.
This is the post I wish I'd had before I started.
---
Hardware
| Component | Spec |
|-----------|------|
| CPU | Intel Core i7-12700 (8P+4E, 20 threads) |
| RAM | 32 GB DDR4-3200 (2× 16 GB, Team T-CREATE CLASSIC CL22) |
| Motherboard | ASRock B660 Steel Legend (Intel B660) |
| Storage | 2× TEAM TM8FP4001T NVMe 1 TB |
| GPU 0 | NVIDIA RTX 3060 12 GB — PCIe 4.0 x16, direct to CPU |
| GPU 1 | NVIDIA RTX 3060 12 GB — PCIe x16 slot through PCH/southbridge (electrically PCIe 3.0 x4) |
| OS | Ubuntu 24.04.4 LTS (kernel 6.17.0) |
| Nvidia driver | 580.126.20 |
One thing worth knowing upfront: GPU1 goes through the B660 PCH, not directly to the CPU. `nvidia-smi topo -m` reports `PHB` for it. This becomes very relevant when choosing split modes (spoiler: `--split-mode tensor` is not your friend here).
Model
GGUF from bartowski/google_gemma-4-26B-A4B-it-GGUF (commit `cb1f989`).
| File | Size | Quant |
|------|------|-------|
| `google_gemma-4-26B-A4B-it-IQ4_NL.gguf` | 13.7 GB | IQ4_NL (4.5 bpw) |
| `mmproj-gemma-4-26B-A4B-it-Q8_0.gguf` | 770 MB | Q8_0 (vision encoder) |
VRAM at runtime: ~6.6 GB GPU0 (54%) + ~11.3 GB GPU1 (94%) with `--tensor-split 6,14`.
llama.cpp build b9029 (`2bacb1eb7`), compiled from source against CUDA 11.8.
---
Final working config
```bash
#!/bin/bash
/usr/local/bin/llama-server \
-m /path/to/google_gemma-4-26B-A4B-it-IQ4_NL.gguf \
--split-mode layer \
--tensor-split 6,14 \
-ngl 99 \
--main-gpu 0 \
--flash-attn on \
--jinja \
--reasoning auto \
-c 131072 \
--cache-type-k q4_0 \
--cache-type-v q4_0 \
--chat-template-file /path/to/chat_template.jinja \
--mmproj /path/to/mmproj-gemma-4-26B-A4B-it-Q8_0.gguf \
--no-mmproj-offload \
--image-min-tokens 560 \
--image-max-tokens 1120 \
--ubatch-size 1200 \
--batch-size 4096 \
--parallel 1 \
--log-prefix
```
A few flags worth explaining:
`--tensor-split 6,14` — this puts 30% of the model on GPU0 and 70% on GPU1. The more balanced `9,11` split is about 2.5% faster decode, but GPU0 ends up at 86% VRAM. With `6,14`, GPU0 sits at 54% (~5.7 GB free). That headroom lets me actually use GPU0 for games or other workloads alongside the model, which was kind of the point. Turns out the bigger imbalance also causes llama.cpp's pipeline parallelism to give up on buffer allocation and silently fall back to a simpler scheduling mode — I was worried about that, but it benchmarks at essentially the same speed so it ended up being fine.
`--cache-type-k/v q4_0` — this is the ceiling on this hardware. Anything higher breaks in different ways (see the experiments table). I tried iq4_nl hoping for better quality and it made decode drop from 65 to 11 t/s.
`--reasoning auto` — lets Gemma 4 use thinking tokens when it wants to. If `max_tokens` is too small, thinking quietly eats the whole budget and you get a truncated reply. Keep `max_tokens >= 2048`, learned that one the annoying way.
---
Benchmark
Using llama-bench (b9029) with `-pg`, which prefills N tokens and then measures decode at that context length. This is worth calling out: `-p N -n 128` and `-pg N,128` are NOT the same thing. The `-p` form measures decode from a near-empty context and gives numbers that look suspiciously good. The `-pg` form fills the KV cache first, which is what actually happens in real use. I initially ran the wrong one and the numbers were flattering enough that I had to double check.
```
llama-bench -m model.gguf -ngl 99 --split-mode layer --tensor-split 6/14 \
--flash-attn 1 -ctk q4_0 -ctv q4_0 -ub 1200 -b 4096 \
-pg 512,128 -pg 32768,128 -pg 65536,128 -r 2
```
Prefill speed:
| Prompt tokens | t/s |
|---------------|-----|
| 512 | 1,832 |
| 32,768 | 2,323 |
| 65,536 | 1,740 |
| 98,304 | 1,413 |
Decode speed (back-calculated from combined `-pg` timings, not directly measured — take these as approximate):
| Context filled | Decode t/s |
|----------------|-----------|
| 512 tokens | ~77 |
| 32,768 tokens | ~52 |
| 65,536 tokens | ~38 |
| ~88,000 tokens | ~36 |
---
Everything I tried that didn't work
| What | Result | Root cause |
|------|--------|------------|
| `--split-mode row` | Hard crash (GGML_ASSERT) | Gemma 4 uses tied embeddings — `output.weight` is a `view_src` of the token embedding tensor. Row-split buffer doesn't handle view tensors → assertion failure in `ggml_backend_cuda_split_buffer_init_tensor`. |
| `--split-mode tensor` | Prefill −65%, decode unchanged | No NVLink. Tensor parallel allreduces across every layer. With GPU1 through the PCH, this completely tanks prefill. Only makes sense with NVLink. |
| `--cache-type-k/v iq4_nl` | Decode 65 → 11 t/s | IQ4_NL's non-linear lookup table needs a larger compute workspace. Pipeline parallel's pre-allocated buffers push GPU0 over the edge → silent fallback to no-pipeline → decode collapses. |
| `--cache-type-k/v q8_0` | Decode 70 → 16 t/s | Q8_0 doubles KV memory bandwidth. RTX 3060 is already bandwidth-limited during decode. Roughly 4x slowdown. |
| `--cache-reuse 256` | No-op | Silently disabled when `--mmproj` is present. Server logs "cache_reuse not supported by multimodal, it will be disabled". |
| `--spec-type ngram-cache` | Decode 65 → 62 t/s | Fine for code or structured output. For natural conversation the verification overhead beats the hit rate. |
---
Gotchas
Vision image tokens — this one burned me for a while. Credit to for the detailed writeup in another thread. Gemma 4 has two parameters for controlling vision budget: `--image-min-tokens` and `--image-max-tokens`. The defaults (40/280) are too conservative. I'm running 560/1120, but seamonn found that 2240 for `--image-max-tokens` actually picks up finer image details than 1120 — possibly because of how llama.cpp tries to fit images between the min/max bounds. The catch: `--ubatch-size` must be >= `--image-max-tokens`, or every single image inference crashes with SIGABRT. The server starts up completely normally (the clip warmup uses a different code path), so you won't know anything's wrong until you actually send an image. If you go to 2240, you'll need `--ubatch-size 2240` or higher, which means more VRAM. On my setup 1120/1200 fits; YMMV.
Reasoning budget — `--reasoning auto` with `max_tokens < 2048` will silently eat your generation budget with thinking tokens and return an empty or cut-short reply. Set it higher than you think you need.
`--split-mode row` crashes, it doesn't just run slowly. Skip it.
Low GPU utilization (30–50%) is normal. Autoregressive decode is memory-bandwidth bound, not compute bound. It will look like the GPUs are doing nothing. They're not.
Pipeline parallel fallback is silent and costs almost nothing. If llama.cpp can't allocate the pipeline buffers it quietly falls back at −1% speed and saves ~3 GB of compute buffer VRAM. You'll see "graph splits: 63" in the logs instead of 3.
---
If you want to build something on top of this
If you're looking to wrap llama-server in your own agent or assistant, pi-mono (https://github.com/badlogic/pi-mono) is worth a look.
---
Hope this is useful. Happy to answer questions.
Thanks to Google for releasing Gemma 4 openly, bartowski for the quants, Mario Zechner for pi-mono, and everyone in this community whose threads I quietly mined for answers. Not thanking Anthropic — their policy changes are kind of why this whole project exists, ha ha.
Posted May 5, 2026, 5:27 PM UTC
14