Realtime voice is the only modality where being right is not good enough — you also have to be right now. A text model can take four seconds and nobody blinks. A voice agent that takes four seconds to start speaking sounds broken. I spent a chunk of last year wiring up bidirectional audio, and most of the hard problems had nothing to do with the model.
The latency budget is a sum, and you don't control most of it
The number users feel is time to first audible byte after they stop talking. That is not one number, it's a chain:
mic capture + encode 15-55 ms
network uplink RTT/2 20-150 ms (mobile: worse, and variable)
