How to Reduce Voice AI Latency: Why Your Retell Agent Feels Slow and What to Fix First (2026)
TL;DR: Callers do not measure latency in milliseconds, they measure it in "is anyone there?" A voice agent feels slow when the gap between the caller finishing a sentence and the agent starting to speak stretches past roughly a second. That gap is the sum of turn detection, the language model, and text to speech, plus anything you bolted on in the middle. The biggest wins are almost never the model: they are tuning end-of-speech detection, cutting the tool calls that block a response, trimming the prompt, and giving the agent something to say while it waits. I build production voice agents for US clients on Retell, n8n, GoHighLevel, and Twilio, and I run VoiceDash, the white-label client portal agencies hand to their clients. Here is the order I actually fix latency in.
The clients who complain about latency almost never use the word. They say the agent "talks over people," or "takes forever," or "sounds like a bad Zoom call." Every one of those is a timing problem wearing a different costume, and every one of them has a different fix.
What the caller is actually measuring
Human conversation runs on gaps of about 200 milliseconds between turns. We are extremely good at noticing when that gap gets longer, and extremely bad at describing it in numbers.
On a voice agent, the gap the caller feels is the whole chain from the moment they stop talking:
- Turn detection. The system decides the caller is done. This is not free, and it is where most of your latency hides.
- Speech to text. The final transcript gets settled.
- The model. Your prompt plus the conversation history goes to an LLM, and you wait for the first usable chunk of a response.
- Text to speech. The first audio bytes get synthesized and start streaming back.
- Network and telephony. The audio actually travels to the caller's phone.
Modern stacks stream and overlap most of these, which is why the total is not just the sum of the worst case. But anything that forces a step to complete before the next one starts, most commonly a tool call, breaks the streaming and you feel every millisecond of it.
Under about 800 milliseconds feels like a person. Around a second and a half feels like a slightly slow person. Past two seconds the caller starts talking again to check the line is alive, which collides with the agent, and now you have a talk-over problem that reads as a completely different bug.
Fix turn detection before you touch anything else
If a client tells me the agent interrupts people, I do not touch the model. I look at end-of-speech settings.
Turn detection decides how long a pause has to be before the agent assumes the caller is finished. Set it too short and the agent jumps in while somebody is thinking of their zip code. Set it too long and every reply feels sluggish even though the model is fast.
The tricky part is that the right value is per client, not per platform. A dental office taking bookings from patients reading a calendar out loud needs more patience than a lead qualification agent asking yes or no questions. Older callers pause longer. People reading an address off an envelope pause a lot longer.
What I do on every build: listen to ten real recordings before touching the setting, and note where the agent cut somebody off and where it left dead air. Then adjust once, in one direction, and listen to ten more. Check the current Retell docs for the exact field names, since these change as the platform ships updates, but the tuning loop is the same on Retell, VAPI, and Bland.
The other half of this is interruption handling. Callers say "uh huh" and "right" while the agent talks. If every backchannel stops the agent mid-sentence, the conversation shreds itself and the client calls it a latency problem. Make sure short filler does not count as an interruption.
Tool calls are where responses go to die
The single biggest latency spike I see in production is a tool call sitting in the middle of a turn. The agent asks for availability, your workflow hits a calendar API, the calendar takes 1.8 seconds, and the caller hears silence the whole time.
Three things to do about it:
Move work out of the call. Most of what agencies wire into the live call does not need to happen during the call. Sending a confirmation SMS, pushing a contact into GoHighLevel, updating a CRM record, notifying the owner: all of that belongs in a post-call webhook, not a mid-call tool. The way I split live tools from post-call automations is in how to connect Retell AI to n8n.
Cover the wait with speech. If a lookup genuinely has to happen live, the agent should say something first. "Let me check the calendar, one second." That single line turns two seconds of dead air into two seconds of a normal human interaction. Most platforms let you attach a filler phrase to a tool, and it is the highest return per minute of work anywhere in a voice build.
Put a timeout on everything. Every live tool needs a deadline and a fallback line. If the calendar has not answered in three seconds, the agent should move on and take a message rather than hanging in silence. An API that is slow once will be down eventually, and a call that dies because a webhook hung is the worst possible way to find out.
If your automation platform is the bottleneck, a cold start on a workflow runner can add a second on its own. Test the actual round trip from the agent, not the API in isolation.
Trim the prompt
Time to first token scales with how much you make the model read. Agency prompts grow: a business description, forty FAQs, three pages of objection handling, a company history nobody asked for. Every turn of every call pays for that.
What I cut first:
- FAQ dumps. Move rarely asked questions to a knowledge base the agent can retrieve rather than carrying them in the system prompt on every turn.
- Redundant instructions. People restate the same rule three ways hoping it sticks. It does not stick better, it just costs more.
- Sample dialogues. One short example beats six long ones.
- Anything about handling situations that never occur on that client's line.
Keep what actually shapes behaviour: identity, the goal of the call, the collection order, escalation rules, and the hard boundaries. The structure I use is in the voice AI receptionist prompt guide, and the escalation half of it is in how to transfer a voice AI call to a human.
Two more model-side levers worth knowing. A smaller, faster model is often the right call for a receptionist that books appointments, because the job is structured and the reasoning is shallow. And long calls get slower as history grows, so if you are seeing the agent degrade at minute eight, that is context, not the network.
Voice choice is a latency decision
Text to speech is not uniform. Some voices are noticeably slower to produce their first chunk than others, and the difference is audible on every single turn.
When I demo voices to a client I am picking two things at once: how it sounds, and how fast it starts. A slightly less warm voice that starts 200 milliseconds sooner usually wins on a phone line, because callers forgive timbre and do not forgive pauses. Test candidate voices on real calls, not in the platform preview, since the preview does not include telephony.
Measure it or you are guessing
"It feels slow" is not a bug report you can act on. Before changing anything, get numbers.
The practical version: pull a handful of recent recordings and time the gaps yourself with the waveform. It is unglamorous and it works. You will usually find the problem is one specific turn, the one with the calendar lookup, and not the agent in general.
Then keep watching it. Latency regresses quietly when someone adds a tool, expands the prompt, or swaps a voice. I check the slowest calls at each monthly client review, alongside the outcome metrics in voice AI client reporting.
One caution: platform dashboards report their own internal timings, which is not what the caller experienced. Your ears on a real recording are the ground truth.
The order I fix things in
When an agent feels slow, I go through this list top down and stop when it sounds right:
- Tune end-of-speech timing to the client's actual callers.
- Confirm backchannels are not counting as interruptions.
- Move every non-essential tool call to post-call.
- Add filler phrases and timeouts to the tools that remain.
- Cut the prompt down to what changes behaviour.
- Consider a faster model for structured, shallow tasks.
- Test the voice for time to first audio, not just tone.
- Re-listen to ten real calls and confirm the gaps closed.
Nine times out of ten the fix is in the first four steps, which is why I stopped starting with model selection.
Where the client sees the improvement
Latency work is invisible on an invoice unless you show it. When I tighten an agent, I want the client hearing the difference themselves rather than taking my word for it.
That is what the portal is for. In VoiceDash the client gets their calls, recordings, transcripts, and usage in a portal with your logo on your domain, scoped to their own data and nothing else. Pull up a call from before the fix and one from after, and the owner hears their own front desk get faster. It connects to your Retell account, is live in under 10 minutes with no code, and plans start at $19/mo with a 7-day free trial. VAPI and Bland support are coming soon; today it is purpose-built for Retell agents.
The bottom line
Latency is a conversation design problem more than an infrastructure problem. The caller is not waiting on your model, they are waiting on a turn detector set for the wrong audience, a calendar lookup nobody covered with a sentence, and a prompt carrying forty FAQs it reads on every turn. Fix those, listen to real recordings instead of trusting a dashboard, and the same stack that felt sluggish starts sounding like a receptionist. That is the bar I hold every build to before onboarding a client onto it.
Want your clients hearing those calls in a portal with your logo on your domain? Start free on VoiceDash or book a demo and I will walk you through the setup on a call.