Source
r/StrixHalo•
Healthy_Effect_9226

Strix Llama: Qwen3.8-Flash-Next (125B MoE) at ~1180 t/s prefill and 33-41 tok/s decode on a Ryzen AI Max+ 395 (Windows installer, open source)

I've spent the past few weeks tuning llama.cpp for exactly one machine and one model, and it's at the point where I'd like other Strix Halo owners to try it and tell me what breaks.

What it is: a patched llama.cpp HIP runtime plus a Windows desktop app (a Jan fork with an installer) for the Ryzen AI Max+ 395 with 128 GB. It serves one model, Qwen3.8-Flash-Next (125B-A6B, Unsloth UD-IQ4_XS, 93.7 GB), with its MTP draft head for speculative decoding. The installer carries the whole runtime: llama-server, the ROCm libraries, a process manager and its own Python. You download the model files, point the app at the folder and press Load. It also serves an OpenAI-compatible endpoint on localhost for other clients.

Numbers on my machine (0.2.3; every one comes with the command that produced it, in docs/results.md):

WorkloadResult
prefill, 95.6K-token prompt1183 t/s (three runs: 1167-1191)
decode, short context41.0 tok/s (MTP, 66% draft acceptance)
decode, 86K context32.6 tok/s (MTP, 63% acceptance)
3 / 4 conversations at once, ~4K tokens each55.7 / 60.4 tok/s summed (1.47× / 1.59× one conversation in the same run)
context256K; optional Q8_0 K/V cache (3.2 GB instead of 6 at 262K)
image inputyes (Qwen3-VL projector)

Optionally, conversation state is kept on disk, so a long chat you come back to isn't processed again: it's read back in about a second (~30 KB per token, capped at 16 GB).

What's different from stock llama.cpp (52 files, also published as one commit on a fork so you can read the diff):

  • IQ3_S weights (52% of this model) now reach the matrix cores. The MMB dequant GEMM didn't accept them before. +5.6% prefill.
  • Expert kernels rebuilt around a gfx1151 fact: the vector ALUs and the matrix cores never run at the same time, so every instruction spent unpacking weights is lost matrix time. The routed gate/up went from 35.9 to 19.9 ms a layer.
  • The rest of a prefill batch fused or rewritten: a 95.6K-token prompt went from ~990 to ~1190 t/s.
  • Sparse attention at decode. At 97K context: 59.0 → 49.3 ms/token.
  • MTP speculative decoding tuned: three draft tokens, and a draft head with its own IQ4_XS output projection. About +60% decode over not drafting.
  • Several conversations at once: drafts sized by concurrency, drafts of equal length, and small-batch kernels for the verify step. The latest release alone took three conversations from 47.7 to 55.7 tok/s summed.
  • First-use stalls removed. hipBLAS loads each GEMM kernel from disk the first time it sees a shape, which cost 0.2-0.4 s in the middle of chats.
  • Two correctness fixes: speculative verification ran without a causal mask, so long answers drifted and stopped early; and image input could abort the server.

Things I learned that might help anyone running HIP on Strix Halo under Windows:

  • Use AMD's TheRock ROCm nightlies (pip wheels), not the HIP SDK 7.1. Same source, same flags: prefill 458 → 735 t/s.
  • The AMD display driver installs its own amdhip64_7.dll into System32, and Windows loads it before anything on PATH. Put the SDK's DLLs next to your exe, or you are running the driver's HIP runtime. I saw different numerics (MTP acceptance 0.48 vs 0.71) until I did.
  • Set the GPU carve to 96 GB. At 64 GB this model spills into shared memory and decode is 28% slower; no software setting gets that back.
  • A load that fails with "bad allocation" while VRAM looks fine has hit the commit limit: RAM plus page file runs out first.

Requirements and limits, up front:

  • Ryzen AI Max+ 395 with 128 GB, and the BIOS GPU carve set to 96 GB. The kernels are compiled for gfx1151 only.
  • Windows 11 only.
  • One model. The attention work is specific to this architecture.
  • About 97 GB of downloads from Hugging Face; the README lists the files with pinned links.
  • Several conversations at once need Concurrent slots above 1 (Advanced settings), and image input needs a single slot, so for now it's one or the other.
  • Several conversations got much better, but that's still where most of the headroom is: each extra verified token costs ~8 ms of a step. That's what I'm working on next.

What I'd love feedback on:

  • Does the installer work on your box (OEM, BIOS version, driver version)?
  • What numbers do you get?
  • Anything that crashes or hangs, with the server log (the app's Developer page shows it).
  • Which model you'd want next.

Links:

Strix Llama is made by Rulith (rulith.ai); we build verifiable execution infrastructure for AI agents.

Posted Sep 25, 2026, 10:53 PM UTC

10
Loading comments…