How we bolted opt-in Apple Metal and NVIDIA CUDA backends onto a pure-C Qwen3-TTS engine — resident fused pipelines, server request-batching, and measuring it all on a Mac mini M2 rented by the hour. Plus the two 'obvious' optimizations we killed with data.
Part of qwen3-tts — a pure C inference engine for Qwen3-TTS.
TL;DR
The engine is pure C and CPU by default. We added two opt-in GPU backends that leave the CPU path untouched:
Apple Metal (make metal) — 0.6B model RTF 1.5 → 0.60 on an M1, 0.36 on an M2 Pro; streaming first-audio in 314 ms.
NVIDIA CUDA (make cuda) — 1.7B model RTF 0.44 on a ~270 GB/s GPU (RTX 4060-class), scaling to ~0.12 on a 4090.
For a take-home assignment from e3 Group, I got a question I couldn't stop thinking about: how fast...
Qwen 3.6 35B-A3B for Local AI in 2026: The 24GB VRAM Line That Gets You 120 tok/s
Same hardware. Same benchmark. Opposite winner depending on model size. Cross-platform inference numbers on Mac M2 Pro Metal,…
I tested Speculative decoding (Multi-Token Prediction, MTP) performance in Qwen 3.6 27B and 35B on an...
My MTP post showed multi-token prediction roughly doubling Qwen3.6-27B's generation on a 3090. A...
Deploy Qwen3.6-35B-A3B-NVFP4 on vLLM: Hopper/Blackwell requirements, serve commands, DGX Spark flags, MTP setup, and what breaks…
