Two things worth sharing from a native-Swift MLX inference project.

# 1. TurboQuant KV compression — V3 quality at V2 speed

The [TurboQuant paper](https://arxiv.org/abs/2504.19874) (Zandieh et al., ICLR 2026) describes a two-pass approach:

* **V2**: Fast linear affine quantization. Hardware-friendly, but degrades quality at 3-bit.
* **V3**: Lloyd-Max non-linear codebooks. Near-optimal distortion, but software dequant is too slow for production.

SwiftLM ports the V3 Lloyd-Max codebooks into the native C++ encoding path, and fuses dequantization into Metal shaders alongside the attention kernel. The result is V3 quality at V2 throughput — no Python overhead, no separate dequant pass.

**K-cache**: 3-bit PolarQuant + 1-bit QJL residual correction = 4.25 bits/dim **V-cache**: 3-bit PolarQuant only = 3.125 bits/dim (QJL provides no benefit here) **Overall**: \~3.6 bits/dim — **measured 4.3× compression confirmed at runtime:**

    [⚡️ SSD Stream] 8515 MB/s | 16374 chunks | avg 0.176 ms/chunk | 🗜 TurboKV 4.3x (17MB saved)
    [⚡️ SSD Stream] 7017 MB/s | 15171 chunks | avg 0.214 ms/chunk | 🗜 TurboKV 4.3x (21MB saved)
    [⚡️ SSD Stream] 8447 MB/s | 17266 chunks | avg 0.178 ms/chunk | 🗜 TurboKV 4.3x (3MB saved)

The QJL (Quantized Johnson-Lindenstrauss) 1-bit residual on K-cache acts as a regularizer against centroid resolution loss in the attention dot-product — particularly relevant for long contexts where V2 degradation becomes visible.

# 2. SSD Expert Streaming for MoE models

MoE models larger than RAM have two failure modes on macOS:

* **VM swapping**: The OS swaps model pages through the VM subsystem. On macOS, this triggers Watchdog kernel panics (SIGKILL) once swap pressure builds.
* **Truncated load**: Reducing context or quantizing further to fit — defeats the point of a 35B+ model.

**SwiftLM’s approach**: mmap the full weight file. For each forward pass, stream only the top-k active expert weight pages from NVMe directly to the Metal GPU command buffer. Non-active expert pages remain on SSD and are never loaded into RAM. The OS page cache handles expert reuse naturally — hot experts for a given prompt stay warm in page cache without any manual management.

This is zero-copy: no intermediate CPU buffer. The Metal driver reads directly from the mmap'd address space backed by NVMe.

**Observed at runtime** (Qwen3.5-122B-A10B-4bit, SSD stream + TurboKV enabled, M5 Pro 64GB):

    [SSD Stream] 670 MB/s   |  1 chunks   | cold start (page cache empty)
    [SSD Stream] 9114 MB/s  | 16842 chunks | avg 0.165 ms/chunk
    [SSD Stream] 7537 MB/s  | 18364 chunks | avg 0.199 ms/chunk
    [SSD Stream] 9245 MB/s  | 19690 chunks | avg 0.162 ms/chunk
    [SSD Stream] 8029 MB/s  | 18075 chunks | avg 0.187 ms/chunk

System Monitor during inference:

|Metric|Value|Notes|
|:-|:-|:-|
|GPU Memory In Use|**2,694 MB**|Only active expert pages in VRAM|
|GPU Memory Allocated|18,769 MB|Full model address space mmap'd|
|macOS Page Cache|**19.6 GB**|Hot experts served from RAM on repeat|
|Available RAM|21.5 GB|Free despite running a 122B model|
|CPU Usage|14.5%|Metal handles inference, CPU idle|
|GPU Renderer|39%||

The page cache behaviour is intentional — on second and subsequent inference passes, the NVMe read drops because the OS already has those expert pages warm in RAM. First-token latency is SSD-bound; generation thereafter is page-cache-bound.

>

**Qwen3.5-122B-A10B-4bit benchmarks on M5 Pro 64GB (SwiftLM/MLX, measured):**

|Config|Prefill|Decode|GPU RAM active|
|:-|:-|:-|:-|
|SSD streaming, 4K context|**25 t/s**|\~0.4 t/s|**2.7 GB**|

Note: At 4,262-token context depth with a 122B MoE, each decode step streams the full active expert set (\~10B params) from NVMe and attends over the entire KV cache. The `predicted_per_second` in the server log is `completionTokens / totalWallClock` (includes prefill) — not the decode rate. Prefill throughput is the more meaningful metric for 122B at large context.

# 3. Qwen3 on iPhone

The iOS app (SwiftLM Chat) runs Qwen3 directly on-device via MLX Swift. Two things made this possible:

* **Flash Attention Metal kernel** — keeps KV cache off the CPU, avoids paging
* **TurboQuant KV compression** — reduces KV memory \~3.5×, enabling longer contexts within the iOS memory budget

On  iPhones 13 Pro (6GB):

* Qwen3-0.6B / 1.7B  — run well

Models download directly from HuggingFace mlx-community, no Mac relay.

Code and build instructions: [https://github.com/SharpAI/SwiftLM](https://github.com/SharpAI/SwiftLM)

Happy to dig into the Metal kernel side if anyone’s interested — the WHT randomization + Lloyd-Max centroid table layout for cache-friendly dequant has some non-obvious implementation decisions.

SwiftLM — Native Swift MLX with TurboQuant KV compression + SSD expert streaming supports Qwen3 on iPhone