Why voice agents fail on the fiftieth call

Every voice agent sounds good in a scripted demonstration. The difference shows on a bad line, with an interruption, at the end of a long day. We built one for ourselves and hit all of it.

We built a voice agent to run our own first-round screening interviews. The reasoning was straightforward: the calls are structured, repetitive and necessary, they consume an interviewer's hour each, and the notes varied depending on who ran them.

The demonstration worked within about two weeks. Getting it to work on real calls with real candidates took considerably longer, and almost none of that time went into the conversation quality that a demonstration shows off. It went into four failure modes that only appear in production.

1. Latency is a design constraint, not a metric

In text, a two-second delay is unremarkable. On a phone call it is a signal, and the signal it sends is that the line has dropped. The caller says “hello?” and starts again, which collides with the agent's response, and the conversation degrades from there.

What matters is not total response time but time to first audio. An agent that begins speaking quickly and continues generating is tolerable; an agent that thinks in silence and then delivers a perfect sentence is not. That single fact constrains every component choice: how speech recognition streams, how early the model starts producing, how the audio is synthesised and buffered.

It is also an argument for keeping inference close to the telephony. A round trip to a hosted API in another region spends budget you do not have.

You cannot optimise latency at the end. Every part of the pipeline gets chosen against the clock, or none of it works.

2. The caller will interrupt, and the agent has to stop

People interrupt constantly in real conversation. They answer before the question finishes, they correct themselves mid-sentence, they say “sorry, can you repeat that” over the top of an explanation.

An agent that continues talking through an interruption is worse than a form, because it signals that nobody is listening. Handling this properly — stopping mid-sentence, discarding the rest of the planned response, and picking up from what the caller actually said — is genuinely difficult, and it is the single thing that most determines whether the interaction feels like a conversation.

The inverse problem is as bad. An agent too eager to yield gets derailed by a cough or a background voice, and a candidate thinking out loud gets cut off mid-thought. Silence is ambiguous: it can mean the person has finished, or that they are considering. Getting that judgment right took us more iterations than any other part of the system.

3. Bad lines are the normal case, not the edge case

Demonstrations happen on good connections in quiet rooms. Production happens on mobile networks, in kitchens, on trains, with accents the recognition model was not tuned for.

What matters here is not accuracy on clean audio, which is a solved problem, but recovery behaviour when audio is poor. Does the agent ask the caller to repeat, and if so how many times before it does something else? Does it degrade gracefully when it catches half a sentence, or does it confidently act on a misheard answer? Does it notice that this call is going badly and change approach?

We tuned recognition for the accents and vocabulary of the people who actually call us, which improved things more than any model upgrade. Generic models are tuned for generic speech, and your callers are not generic.

4. A confident wrong answer costs more on a call

In text, a reader can pause on a claim that looks wrong, check it, or scroll back. On a call the answer is spoken once, at conversational speed, with no citation and no opportunity to verify. If it is wrong and delivered confidently, the caller acts on it.

This changes where the boundaries have to sit. A voice agent needs a narrower scope than a chat agent doing the same job, clearer instructions to admit uncertainty, and a faster path to a human. “I am not sure about that, let me pass you to someone who is” is a good outcome on a call, and it needs to be as easy for the agent to reach as any other response.

The reason we built it on ourselves first

Every one of these problems cost us something to discover: a candidate's poor experience, a call that had to be redone, a week of work on a component we had assumed was finished. That is an acceptable price to pay on our own hiring and an unacceptable one to pay on a client's customers.

Which is why the product exists and why it sits on this site as proof rather than as a case study we were given. Voice fails in production, not in the demo. We have already had those calls fail.