Compression sidecar

I wanted to share this because I haven't really seen anyone talking about it and i figured it may help those in my situation. I've been progressively running larger and larger models, adding additional GPUs to handle them. I've noticed, the larger I went, the less useful they were due to constantly needing to compress the context. The larger the model, the more i hated compression.

When i reached Kimi k2.6 i was both the most happy and the most frustrated. K2.6 was absolutely killing every task and when comparing to ChatGPT 5.5 i found the two were so close as to basically be equivalent in both coding and hardware configuration and design. But as i moved deeper into a complex problem, something that will definitely take well over 10 prompts to get done, kimi would become unusable, it would spend tens of minutes just chunking through context. By the 10th prompt a simple question could take over an hour.

I thought that adding more hardware would alleviate some of the problem, all it alleviated was my savings account. What did improve it drastically was simply setting up a smaller sidecar just to focus on compression. In my case i added Qwen3.5 8b. When comparing having Kimi do the compression and having Qwen compress, it was the difference in giving up after 15 minutes and tens of seconds.

I tested a couple smaller models like llama3.1 along with the Qwen2.5. I saw very little difference however qwen far outperformed when taking into account vram usage.

| | Qwen Q8_0 | Llama Q4_K_M |

|-----------|-----------|--------------|

| Speed | 9.2s | 7.4s |

| VRAM | 13 GB | 21.7 GB |

| Speedup | Baseline | ~20% |

| VRAM cost | Baseline | +67% |

When dropping qwen down to q4 i saw the largest improvement

| Qwen3.5-9B Q4_K_M | Qwen3.5-9B Q8_0 |

|-------------------|-------------------|-----------------|

| Prompt tokens | 42,912 | 42,912 |

| Completion tokens | 500 | 500 |

| Total wall time | 13.8s | 84.0s |

| VRAM usage | 9,882 MiB | 13,064 MiB |

And when i added a second GPU it was even better!

| Configuration | Prompt tokens | Completion tokens | Total time | VRAM per GPU | |---------------------------|---------------|-------------------|------------|-------------------|

| Qwen Q8_0 (single GPU) | 42,912 | 500 | 84.0s | 13,064 MiB |

| Qwen Q4_K_M (single GPU) | 42,912 | 500 | 13.8s | 9,882 MiB |

| Qwen Q4_K_M (2-GPU split) | 42,912 | 500 | 6.0s | 5,494 + 6,036 MiB |

So i settled on qwen3.5 8b q4 as the sidecar processing my hermes compression. Since making the final decision i compressed about 15 times. Hermes was making fantastic/smart choices. Compressions stayed well under a minute even when compressing as many as 120k of cache. Its an absolutely wonderful experience now that i can enjoy Kimi the way it was intended.