The first thing I check when a client tells me their AI voice agent "feels slow" is not the model. It is almost never the model. I have swapped a faster model into a sluggish agent and had callers notice nothing at all, because the seconds they were feeling were never being spent where everyone assumed.
If you have ever profiled a slow page and found the real cost was a render blocking script rather than the function you spent all week optimizing, you already understand this post. Voice just makes the mistake more expensive, because a phone call has no spinner. No loading state, no skeleton UI, no way to signal "still working" other than talking. Silence is the only progress indicator you get, and to a human on a phone, silence means the line dropped, the other person is confused, or nobody is listening. All three make callers talk over the agent, repeat themselves, or hang up.
I build production voice agents on Retell wired into n8n, GoHighLevel and Twilio. Before that I spent years in Unity and AR, where a late frame was a safety problem rather than a conversion problem. That is where I learned to treat latency as a property of the whole chain rather than a number attached to one component.
