Pete Hanlon, CTO of Moneypenny. Moneypenny handles outsourced phone calls, live chat and digital comms for thousands of companies globally.gettyYou are on the phone with an automated system. You pause for a second to think, and the AI jumps in before you’ve finished. You start speaking again, and now you are both talking at once. You both stop. Then comes a silence that lasts a beat too long, while neither of you is sure whose turn it is.The rhythm is broken, and once it breaks, the conversation rarely recovers.By the end of the call, you are irritated, even though nothing obviously went wrong. The system heard you. It may even have understood you. But the conversation did not feel natural.That failure is not just a rough edge. It is a turn-taking problem, and turn-taking remains one of the most underappreciated engineering challenges in voice AI.Getting it wrong is one of the fastest ways to lose a caller's trust.The Problem With SilenceIn a chat system, turn-taking is straightforward because there’s a send button. You type. You send. I read. I reply. The boundary between your turn and mine is explicit.Voice has no send button.A voice AI system has to infer when you’ve finished speaking from the sound of you stopping, the words you used before you stopped and the conversational context around that pause.None of those signals is simple. A pause after a complete sentence means something different from a pause halfway through one. A trailing “and” carries a different signal from a trailing “so.” A half-second silence might mean “I’m finished.” It might mean “I’m thinking.” It might mean “I’m upset and waiting for you to help.”To a microphone, those silences can sound almost identical. To a caller, they mean completely different things.That creates the tension at the heart of every voice interaction. Respond too quickly and the system talks over the caller, clipping the most important part of the sentence and forcing them to start again. Respond too slowly and the conversation feels awkward or the caller starts wondering if anyone is there. The window that feels natural is incredibly narrow, and it changes with every caller.​And in customer service environments, broken rhythm increases handling time, creates repetition and it makes the system feel less capable than it actually is.​Why Speed Doesn’t Solve The ProblemEven the best turn-taking judgement is worthless if the underlying system can’t keep pace.From the moment a caller stops speaking, the system has to detect the turn may have ended, process what was said, generate a response and begin delivering it. All inside a window measured in milliseconds.If any part of that chain is slow, the quality of the judgement becomes irrelevant because the response arrives too late anyway. A good decision delivered half a second late is indistinguishable from a poor one.In voice AI, timing is part of the experience, not a technical detail. Too often, the entire problem gets reduced to a single setting: Voice Activity Detection, essentially a silence threshold configured once and deployed everywhere.But that’s where turn-taking begins, not where it ends. Better systems measure silence and try to understand what that silence means. They weigh words, pacing and intonation to judge whether the thought is actually complete. A trailing “and” should buy the caller more time than a crisp “thank you.”They also need barge-in handling, so callers can interrupt naturally without collapsing the interaction, and back-channeling; I'm talking about those small signals like “mm-hmm” or “right” that show someone is listening without taking over the conversation.A system built only around silence thresholds has solved the simplest part of the problem and mistaken it for the whole.Faster Isn’t The Same As SmarterThere’s a growing assumption that speech-to-speech models will make this challenge disappear.They genuinely help. By processing audio directly, rather than running speech-to-text, then a language model, then text-to-speech, they reduce latency and preserve conversational cues that text often strips away, like rising intonation, trailing breaths, changes in pace or signs of frustration.But reducing latency isn’t the same as solving turn-taking. A faster pipeline still has to answer the same questions.Is this pause hesitation or completion? Should the system respond here or wait? Is the caller finished or simply thinking?Whilst speech-to-speech models compress the time between hearing and responding, they do not remove the need for judgement, and that’s the risk.Speed can start masquerading as intelligence, but if a system simply interrupts callers faster than before, it isn’t progress. It’s just failure at lower latency.​The Difference Between Hearing And ConversingThis is why turn-taking is the real question worth asking. If you want to know how seriously a vendor takes voice AI, don’t just ask about the model.Ask how the system handles hesitation, interruption, emotional speech and mid-sentence changes of direction. Those are the moments that reveal whether it was built for real conversations or simply for clean demos.At Moneypenny, millions of customer conversations have taught us that turn-taking can’t be treated as a configuration setting. It has to be engineered into the interaction itself through adaptive silence thresholds, barge-in handling, real-time performance architecture and systems designed around how people actually speak. It’s also why we believe the strongest customer experiences come from blending AI efficiency with human expertise, with each stepping in where they add the most value.The next time you evaluate a voice AI platform, call it. Not with a perfect script and a quiet environment, but the way a real caller would. Pause halfway through a sentence. Change your mind. Interrupt it. Trail off. Start again.If the conversation survives all of that and still feels natural, you’ve found a system that takes turn-taking seriously.If it doesn’t, everything else it has built is sitting on a broken foundation.Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?