I built a Rust inference framework that runs Qwen3.5 2B with VL support 10x faster than PyTorch on Apple Silicon — and it supports TTS, ASR, OCR, and GGUF out of the box

Hey r/Rust (and r/localLLaMA),

After years of wrestling with C++ inference backends, I built Crane 🦩 — a high-performance LLM inference framework powered by Rust's Candle (Hugging Face's ML framework). The goal was simple: fast inference without the C++ headache.

Why should you care?

🚀 Blazing fast — 50x faster than native PyTorch on M1/M2/M3 for Qwen3-VL 2B 🦀 Pure Rust + Candle — one dependency, no Python, no C++ 🍎 Metal GPU — first-class Apple Silicon support 🤖 CPU / CUDA / Metal — unified codebase for all hardware 📦 GGUF out of the box — drop in any llama.cpp GGUF file, auto-detected 🗣️ TTS / ASR / OCR / VAD — not just LLMs, a full AI pipeline ready to go 🌐 OpenAI-compatible API — swap in any OpenAI SDK, works instantly

What's supported

• Qwen3.5 (0.8B–30B+) with hybrid GDN attention
• Qwen2.5 (0.5B–72B)
• Gemma 4 (text + vision)
• Hunyuan Dense
• Qwen3-TTS (12Hz, 24kHz, voice cloning)
• Voxtral-4B-TTS (20 voices, 10 languages)
• Moonshine ASR + Silero VAD
• PaddleOCR v6

The differentiator: Crane isn't a low-level SDK. It ships with ready-to-run AI apps — chat, VLM, OCR, TTS, ASR — all out of the box. You can build your own or use what's there.

Recent highlights

• Qwen 3.5 GGUF loading with auto-detected hybrid GDN blocks (no sibling files needed)
• In-situ quantization (--quant q4k|q8_0) — quantize at load time, no conversion step
• Qwen 3.5 Vision end-to-end with OpenAI-style image_url support
• Ornith tool-calling agent support

Repo: https://github.com/lucasjinreal/Crane

Would love feedback from the community — especially CUDA on Linux and anyone interested in contributing model support. Submit your models and make them usable for everyone! 🦩

Want me to adjust the tone (more technical / more casual), target a specific subreddit, or add any particular details?