Most people think building an AI voice agent means writing a clever prompt. I build these for a living, and I can tell you the prompt is maybe an hour of the work. The other week disappears into two places: wiring up everything the agent touches, and testing it against the twenty ways a real caller will break it.
So I built a pipeline that points one AI coding tool at each of those problems. Claude Code generates and wires the agent from a spec. Comet, an AI browser automation tool, runs it through dozens of messy call scenarios before a human ever picks up the phone. This post is how that loop actually works, and where it still needs me.
Why the build loop is slow (and it is not the prompt)
When you picture building a voice agent, you picture the prompt. That is the easy part. The slow part is everything around it.
A production agent for, say, a car garage is not one artifact. It is a conversation flow, a set of custom functions that hit your automation layer, calendar and CRM wiring, a telephony number with A2P registration, and a pile of edge-case handling that only shows up when someone calls in angry with a dog barking in the background.
