ea_man

You know that when you start Pi you have to process the initial prompt, that takes some time when you use the slow dense QWEN 27B (that's the very reason why you use Pi instead of Cloud Code!), then you start to add extensions, tools, your append.md and whatever...
Well now that got big, like 20k big and it does bother.

So let's cache the "initial prompt" PP, so that when you start an new pi session: TA-DA! Instant ready, jolly good.

Well if you tried to do that with Pi, llama.cpp and QWEN 3.x hybrid KV all kind of things step in your way to prevent that, so many that I won't even start to count I'll just tell you what to do:

1. dwl and install [this Pi extension](

Today I finally hit a wall with QWEN 3.8 dense 27B, Q6_X_L with KV cache q8_0 temp 0.85 with medium reasoning effort, producing over thinking, failing to follow precise instructions, re evaluation in loops data already seen, "Actually…/Wait…/Hmm…", meta-reasoning, theorizing about user intent and you know what else...

So I stopped and asked GPT Sol and QWEN to read the reasoning traces of my old Pi coding sessions, identify recurring misbehaviors and propose an appropriate APPEND_SYSTEM.md to tame those in Pi (place that in ~/.pi/agent/APPEND_SYSTEM.md and it will follow the initial prompt).

So for those who wanna check out

Today I finally hit a wall with QWEN 3.8 dense 27B, Q6_X_L with KV cache q8_0 temp 0.85 with medium reasoning effort, producing over thinking, failing to follow precise instructions, re evaluation in loops data already seen, "Actually…/Wait…/Hmm…" and you know what...

So I stopped and asked GPT Sol and QWEN to read the reasoning traces of my old Pi coding sessions, identify recurring misbehaviors and propose an appropriate APPEND_SYSTEM.md to tame those in Pi (place that in ~/.pi/agent/APPEND_SYSTEM.md and it will follow the initial prompt).

So for those who wanna check out:

• https://store.piffa.net/lm/APPEND_SYSTEM.md
• [https://sto

So I want to be able to start Pi instantly by reloading a saved KV slot (llama.cp).

What I come out as a proof of concept:

# to regenerate start.bin initiate a pi session, save the slot, copy to start.bin
SLOT_FILE="start.bin"
SESSION_ID="01a06cea-4e97-7378-84e6-86c3274302da"
cp  /home/eaman/.pi/agent/sessions/--home-eaman-test--/start_session.jsonl /home/eaman/.pi/agent/sessions/--home-eaman-test--/2026-09-04T14-38-07-702Z_01a06cdb-0356-7001-949d-209024b066d8.jsonl

# Discard changes from the previous run and restore the session that matches
# start.bin.

curl --fail --silent --show-error \
  -X POST "http://localhost:8080/slots/0?action=restore" \

I've made a modification of llama.cpp MTP for people that want to run models like QWEN 27B on 16GB and similar setup, the focus is reducing the memory cost of MTP allowing more speed for less ctx cost.

MTP Mode	Maximum Draft (n)	Available Context	TG (t/s)
Standard	2	72,192	39.53
MTP Compact Rollback	5	77,312	46.39

On this example of a (well tuned!) IQ4 running on 16GB you get some +5k ctx and enjoy 17.35% increase on token generation.

With MTP the more speculative tokens you generate (n-max) the more the speed increase chance yet you pay a price in vRAM for those (rollbacks in case the latest token are not accepted) and that reduces the ctx

I was reordering my launch scripts for QWEN 3.8 27B on my 16+12GB setup, those are used GPU cards worth some less than 500s both on the used market giving 28GB on RDNA2.

In case someone could be interested I'll give you a summary:

* For quick shots: Qwen3.8-27B-IQ4-MIX.gguf , TG speed ~47t/s for 10k generation, ~70k usable ctx

* Daily work: Qwen3.8-27B-Q6_K_L.gguf , TG speed at 72k (not zero, at mid session) ~20t/s, 140K ctx with KV cache q8/q8.

I'll give you the full table:

#  | Model / Quant | Backend | Ctx Avail | KV         | TG (t/s) |
---|---------------|---------|-----------|------------|----------|
 1 | 27B Q6_K_L    | Vulkan  | 107 520   | q8_0/q8_0  |   20

I've made some optimizations for llama.cpp in order to save vRAM for models like QWEN 27B and 35B A3B, in order to have more ctx available on 12GB / 16GB and multi GPU setup, I run a 6800 + 6700xt.

1.--hip-fa-force-vec on|off  

Forces the supported quantized-KV HIP Flash Attention VEC path. Enables larger context    at the cost of slower prompt processing speed.  Default is off (ROCm/HIP only).
Forcing VEC allows a substantial reduction in vRAM usage at the cost of Prompt Processing speed, TG speed remains almost the same.

QWEN 3.8 27B IQ4 on 16GB:
# ROCm ctx VEC forced: 89088 q5_1,  96512 q5_0, 116480 q4_0, TG speed: 42.35 t/s, PP for 32k: 145.12 t/s
# ROCm ctx VEC off 

Available context length with and without the patch:

| Model: QWEN 27B | ROCm stock | patched | Vulkan stock | patched |

| :--- | :--- | :--- | :--- | :--- |

| IQ4\_XS Pure, single 16GB GPU | 19.456 | 76.032 | 68,352 | 78,592 |

| Q6\_K\_L on 16GB + 12GB | 64,256 | 149,248 | 68,864 | 151,296 |

The issue is that llama.cpp overestimates the memory needed for MTP compute-buffer/scheduler allocation during auto-fit, that leaves much less ctx available to the user than what actually needed by MTP. This patch stops the fitter from throwing away context based on an inflated MTP memory estimate

Patch, launch scripts used for llama-server and raw logs: [https://store.piffa.net/lm/bu