Running on-device LLM in Unity Android — 523s → 9s with llama.cpp + Adreno OpenCL (79x speedup)
Been building a roguelike RPG where an on-device LLM generates dungeon content every 5 floors — mob names, dialogue, boss patterns — no server, fully offline.
The journey to get usable inference speed was rough:
| Approach | tok/s | Notes |
|---|---|---|
| ONNX Runtime CPU | 0.21 | 523s per generation |
| ONNX + QNN HTP | 0.31 | 3/363 nodes on NPU (INT4 unsupported) |
| LiteRT-LM GPU | — | Unity renderer killed available VRAM |
| llama.cpp Adreno OpenCL | 16.6 | 9s per generation |
Final stack: Qwen3-1.7B Q8_0 (1.8GB) + llama.cpp OpenCL on Snapdragon 8 Gen 3.
One counterintuitive finding: on Adreno OpenCL, Q8_0 is faster than Q4_0. Lower quantization introduces dequantization overhead on the GPU that actually slows things down.
Unity integration needed a C wrapper (unity_bridge.c) — direct P/Invoke of llama.h structs causes SIGSEGV due to layout mismatch.
Posted Apr 6, 2026, 1:27 PM UTC
35