For a take-home assignment from e3 Group, I got a question I couldn't stop thinking about: how fast can text-to-speech start speaking if you refuse to launch a thousand CUDA kernels?
The answer, on an RTX 5090: the megakernel-backed decoder reached ~25 ms time-to-first-chunk, running at RTF 0.12 (it generates audio tokens ~8× faster than real time). Repo: e3-megakernel-tts-takehome.
Honest scope note before anything else: this was a take-home focused on the talker-decode path. The codec stage was partly stubbed, so I make no claims about production-quality end-to-end audio — the interesting engineering (and the numbers) live in the decoder.
The idea: one kernel that never exits
A normal LLM/TTS decode step launches a long chain of kernels — embedding lookup, attention, MLP, norms, sampling — and every launch pays fixed overhead. For big-batch training that overhead amortizes to nothing. For latency-critical, batch-of-one decoding it is the cost: microseconds of launch latency and pipeline bubbles between every op, dozens of times per token.
