My pipeline was fast enough to interrupt people. Here is what I changed, what it cost, and the second bug that made calls go deaf without anyone noticing.
I run a voice AI sales training platform. Users practice cold calls against an AI buyer, in real time, over WebRTC. The stack is Pipecat for orchestration, LiveKit for transport, Deepgram Flux for STT, Cartesia Sonic for TTS, and a small fast model over OpenRouter for the conversation.
In July I had a pipeline I was proud of: p95 of about 880 ms from confirmed end of turn to the first audio byte coming back. Then I listened to real calls, and the AI buyer kept talking over people.
Not at natural turn boundaries. Mid-sentence. A rep would say "so what we do is...", pause half a second to think, and the buyer would answer the unfinished thought.
The pipeline was working exactly as designed. That was the problem.
