Dual Strix Halo: No Frankenstein setup, no huge power bill, big LLMs

Processing img 0vkkozxhyrcg1...

Software on Strix Halo is reaching a point where it can be used, even with networking two of these PCs and taking advantage of both iGPUs and their 256GB of quad channel DDR5-8000 memory. It requires some research still, I can highly recommend the Strix Halo wiki and Discord.

On a single Strix Halo you can run GPT-OSS-120B at >50tokens/s.

With llama.cpp and its RPC feature I can for example load Minimax-M2.1 Q6 (up to 18tokens/s) or GLM 4.7 Q4 (only 8 tokens/s for now).
I'm planning on experimenting with vLLM and cerebras/DeepSeek-V3.2-REAP-345B-A37B next week.

Total cost was 3200€^(*) including shipping, VAT and two USB4 40GBps cables.

What's the catch? Prompt preprocessing is slow. I hope it's something that will continue to improve in the future.

^(*) prices have increased a little since, nowadays it's around 3440€