

> 🚨 **NOTICE:** This article was edited and refined with the help of an LLM, as English is not my native language. I will also be using an LLM to respond to technical comments in this thread. You can accuse me of whatever you like, but the source configurations, deployment scripts, architectural findings, and inference logs are 100% real, captured on my physical hardware, and completely reproducible. Toxic users and armchair critics are kindly asked to leave this thread immediately. I have zero tolerance for baseless claims. Let's speak the language of computational physics.

---

Hi everyone! 

Following up on my previous post: https://www.reddit.com/r/LocalLLM/comments/1w46tzh/qwen38flashnextuncensored_125b_moe_running_on_a/

I decided to push further to achieve rock-solid prompt processing and token generation metrics. Today, I'm excited to share my results.

This post outlines the practical experience of deeply optimizing the inference pipeline for a massive Mixture-of-Experts language model: **Qwen3.8-Flash-Next-Uncensored (176.94B, i1-Q4_K_S quantization, 104 GB file size)**. The benchmarking rig is running **Melawy Linux** (powered by the XanMod kernel) on an **AMD Ryzen 7 5700X (8C/16T, 32MB L3)**, **128 GB of system RAM** (with zswap enabled), and a brand-new discrete **AMD Radeon RX 9070 XT with 16 GB VRAM**, running under **ROCm 7.2.4**.

The core challenge was driving a model that outsized the GPU's video memory by nearly 7 times, ensuring it ran at its absolute physical limit with stable, stutter-free performance.

Through days of rigorous profiling, I established three critical architectural conclusions that directly contradict many standard optimization guides found online.

---

## 1. The Speculative Decoding Paradox: Why MTP Tanks Performance in Hybrid MoE
Built-in speculative decoding mechanisms (`--spec-type ngram-mod` or custom MTP drafters hooked up via `-md`) usually yield a 1.5–2x speedup on traditional Dense models. However, in a hybrid MoE split-inference scenario—where the expert layers run on the CPU and the core attention blocks sit on the GPU—these mechanisms proved **highly detrimental**.

Activating MTP introduced a severe hardware execution bottleneck—a *Pipeline Stall*:
1. The algorithmic drafter guessed a sequence of tokens using the CPU cores.
2. The engine abruptly dispatched this batch to the GPU to verify the base attention layers, completely saturating the PCIe bus.
3. During this brief verification window, the CPU threads suddenly lost their compute load and went to sleep, waiting for the GPU to return control.
4. Once the GPU finished, the CPU cores had to abruptly "wake up" and fetch the massive MoE expert matrices from system RAM all over again.

In the logs, this hardware thrashing appeared as disastrous drops in instant throughput (`tg_3s`) down to **2.26 t/s**, followed by erratic performance spikes. **The Fix:** Completely disable MTP/speculative modules and stick to strict linear inference. Token generation instantly stabilized into a flat, predictable line between **14.55–14.71 tokens per second** across the entire context window.

---

## 2. Hard Hardware Thread Pinning (`--cpu-strict 1`) — A Silver Bullet for MoE
Offloading expert blocks to system memory (`--cpu-moe`) makes the execution pipeline incredibly sensitive to the state of the CPU's L3 cache. The default Linux Completely Fair Scheduler (CFS) tends to bounce threads across physical and virtual cores haphazardly, invalidating the CPU cache on every context switch.

The native topology manager built into `llama.cpp` solved this perfectly:

```bash
  --threads 8   --threads-batch 16   --cpu-range 0-15   --cpu-strict 1 
```

* We allocated exactly 8 execution threads and pinned them strictly to the first half of the CPU topology (the physical cores).
* The `--cpu-strict 1` flag explicitly forbids the Linux kernel from migrating these threads. Consequently, the active MoE expert weights remained resident in the processor's 32 MB L3 cache.
* The remaining SMT companion threads (cores 9–16) stayed entirely free (hovering around 2–8% utilization). They seamlessly absorb low-level OS overhead: Btrfs file system writebacks, the SSH daemon, and the `lactd` fan control service, keeping the desktop completely lag-free.

---

## 3. The Pitfalls of High Real-Time Priorities: Why `Nice=-5` Beats `-20`
Attempting to give the inference server maximum Real-Time priority (via flags like `--prio 3` or `Nice=-20`) caused a paradoxical failure: during a cold boot, the server completely locked up the terminal and SSH session for a couple of minutes.

The issue stems from high-priority `llama-server` worker threads completely starving the low-level Linux kernel I/O tasks. While the engine was initializing, its worker threads overrode the kernel threads trying to execute file mapping via `mmap` from the SSD and allocate RAM pages. The inference engine essentially prevented the operating system from reading its own model files, triggering a severe I/O Lock.

The **sweet spot** turned out to be **`Nice=-5`**. This gives the inference server a solid scheduling advantage over ordinary user space apps during compute cycles but leaves enough breathing room for kernel-level I/O operations, allowing the 104 GB model to map into memory instantly.

---

## Real-World Performance Metrics:
* **Prompt Processing (Prefill):** Consuming a chunk of **26,081 tokens** took just **135.73 seconds**—maintaining a steady **192.15 tokens/sec**! This represents an excellent utilization of `--threads-batch 16` on AMD silicon.
* **Token Generation (Eval):** A completely flat, unwavering line at **14.60 tokens/sec** deep into the context window.
* **Resource Allocation:** VRAM is packed perfectly (`15.85 GB / 15.92 GB`), keeping the PCIe bus completely clear of unnecessary memory thrashing. The GPU power draw drops down to a modest **78W** while waiting for CPU expert execution, keeping core temperatures at a chilly **32°C**.

I have attached my finalized, production-grade systemd `.service` file configuration with the exact launch parameters to this post. Configure your deployment rigs with care!

---

### Authors & Contributors:

* 👩🏼 **Valeria Fadeeva** — Main Developer and Founder of **Melawy Linux** (Arch-based distribution).

Supported by Neural Network Models:

1. **Google Gemini 1.5 Pro** — Assistance with material structure, cross-referencing documentation, and compiling disparate online data.
2. **Qwen-3.8-Flash-Next-Uncensored (i1-Q4_K_S)** — Executing real-world benchmarking tests by generating complex Rust code directly on the target rig.

```ini
[Unit]
Description=Llama.cpp Server for Qwen3.8-Flash-Next-Uncensored (i1-Q4_K_S): port 11441
Wants=network-online.target
After=network.target network-online.target

# Prevention against infinite restart loops
StartLimitIntervalSec=300
StartLimitBurst=3

[Service]
# Control over hung processes in case of disk I/O blocks
TimeoutStopSec=60
# Forceful periodic health-cleanup (optional)
RuntimeMaxSec=10800

# Canonical execution under local user
User=lera
Group=lera

# Low-level environment wrapper for ROCm/HIP and compiler under gfx1201 (AMD Radeon RX 9070 XT)
Environment="AMD_LOG_LEVEL=0"
Environment="ROCM_PATH=/opt/rocm"
Environment="HIP_PATH=/opt/rocm"
Environment="LD_LIBRARY_PATH=/opt/rocm/lib"
Environment="PATH=/opt/rocm/bin:/usr/local/bin:/usr/bin:/bin"
Environment="ROCM_ALLOCATOR_STRATEGY=2"
Environment="HSA_ENABLE_SDMA=1"
Environment="ROCBLAS_TENSILE_WARMUP=0"
Environment="HIP_VISIBLE_DEVICES=0"
Environment="HIP_PLATFORM=amd"

# Path to the directory containing model files
WorkingDirectory=/data/llama/models/mradermacher/Qwen3.8-Flash-Next-Uncensored-i1/

# Protections against swapping context and file descriptor expansion
LimitMEMLOCK=infinity
LimitNOFILE=65536

# Pure MoE execution command with strict CPU thread pinning and no MTP overhead
ExecStart=/usr/bin/llama-server   -m Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_S.gguf   --mmproj Qwen3.8-Flash-Next-Uncensored.mmproj-f16.gguf   --image-min-tokens 1024   --jinja   --no-skip-chat-parsing   --reasoning on   --reasoning-preserve   --reasoning-format auto   --cpu-moe   --n-gpu-layers 99   -fit off   --threads 8   --threads-batch 16   --cpu-range 0-15   --cpu-strict 1   --cpu-range-batch 0-15   --cpu-strict-batch 1   --batch-size 2048   --ubatch-size 512   --parallel 16   --ctx-size 262144   --context-shift   --cont-batching   --kv-unified   --cache-prompt   --cache-ram 8192   --cache-idle-slots   --cache-type-k f16   --cache-type-v f16   --flash-attn on   --temp 1.0   --top-k 20   --top-p 0.95   --min-p 0.00   --repeat-last-n 512   --repeat-penalty 1.00   --presence-penalty 0.00   --host 0.0.0.0   --port 11441   --reuse-port

Restart=always
RestartSec=10s

# Priority fine-tuned by real-world tests (optimal throughput, no I/O choking)
Nice=-5
OOMScoreAdjust=-500

MemoryHigh=115G
MemoryMax=125G

[Install]
WantedBy=multi-user.target
```


How to Run a 176B Model (104 GB) on 16 GB VRAM Without Freezes: The Triumph of Pure MoE Architecture and Core Isolation