Every part of a speech→LLM pipeline is fast enough on its own. Put them in a row and you get three seconds, which is far too slow when a human is waiting for you to say something.
I build a desktop overlay that listens to the other side of a video call, transcribes it, and streams an answer. The budget I care about is time from the speaker finishing a sentence to the first token on screen. Here is where that second and a half went, and what actually moved it.
The naive pipeline
mic/loopback → PCM → WebSocket STT → final transcript
→ "is this a question?" classifier → LLM → stream
