What is voice agent latency?
Voice agent latency measures the conversational gap between a user’s turn and the agent’s reply. For a useful end-to-end measurement, start at the last speech sample of the completed user turn and stop at the first audible sample of the agent’s response, both at the user endpoint.
End-to-end response latency = first audible agent response time − end of caller speech time.
In a phone deployment, that endpoint is the caller’s handset. In a browser deployment, it is the client’s audio output. A server timestamp showing that audio was generated or sent measures an earlier boundary. Label it accordingly if endpoint playback cannot be measured.
Response latency, inference latency, and network latency
These measurements answer different questions:
Metric | Measurement boundary | What it tells you |
End-to-end response latency | Caller finishes → caller hears agent | The conversational wait |
End-of-turn detection delay | Speech ends → system accepts the turn as complete | How long the agent waits before responding |
STT finalization delay | Relevant speech ends → final transcript becomes available | Whether the text is ready for downstream use |
LLM time to first token, or TTFT | Model request starts → first output token arrives | How quickly model output begins |
TTS time to first byte, or TTFB | Synthesis request starts → first audio byte arrives | How quickly audio generation starts delivering data |
Tool latency | Tool request starts → validated result is available | The wait for an external operation |
Barge-in stop latency | User starts interrupting → agent audio stops at the endpoint | How quickly the agent yields |
LiveKit exposes per-turn and component metrics that help separate these stages. Its pipeline estimate combines end-of-utterance delay, LLM TTFT, and TTS TTFB; endpoint measurements are still needed to establish what the caller actually hears.
Inference latency is the time spent running a model on an input. Depending on the provider, a reported figure may exclude queues, transport, or application work. LLM latency also needs a boundary: first token and complete answer are different measurements. A tool-call token or punctuation may arrive before there is anything useful to speak.
Network round-trip time measures a journey to a remote point and back. It does not include the full reasoning and speech-generation workflow, and it should not be used interchangeably with the conversational gap.
Why is latency important for AI voice agents?
Timing helps people decide whose turn it is to speak. An unexplained pause can prompt a caller to repeat the question just as the agent starts answering. Responding too early can interrupt a name, address, or correction.
Research by Stivers and colleagues examined turn-taking across 10 languages and found response distributions peaking within 200 ms of question completion. This is evidence about human conversation, not a universal 200 ms requirement for voice AI.
For a production voice agent, test whether slower turns coincide with more repetitions, abandoned calls, failed interruptions, or incomplete tasks. Treat these as outcomes to measure in your own deployment. A lower latency number has limited value if it comes from misunderstanding callers or skipping a required confirmation.
What is a good latency for a voice agent?
A good latency target is a measurable response objective for a defined workflow. Simple question-and-answer turns, account lookups, and multi-step transactions need different expectations.
For an initial evaluation, use the following proposed targets, then revise them using representative calls and user feedback:
Scenario | Proposed objective | Acceptance condition |
Simple turn with no external tool | p50 ≤ 800 ms; p95 ≤ 1,500 ms from speech end to audible response | No unacceptable regression in turn detection or answer quality |
Lookup or transaction | Track acknowledgment time and verified-answer time separately; set the answer objective from the dependency budget | No claim of success before a validated result |
User interruption | Establish a separate endpoint audio-stop objective from baseline testing | Agent yields reliably and handles false interruptions correctly |
Noisy, multilingual, or hesitation-heavy speech | Set segment-specific objectives | Callers can finish their turns and critical entities remain accurate |
The p50 is the median: half of measured turns are at or below it. The p95 captures the threshold below which 95% of measured turns fall. Report the sample count, time window, and excluded events with both.
What published latency numbers actually mean
Published number | Source and date context | Correct interpretation |
Response-distribution peak within 200 ms across 10 languages | Stivers et al., 2009 | Human turn-taking research, not an AI service-level agreement |
300–500 ms endpointing setting for speech with mid-thought pauses | Deepgram configuration guidance, accessed September 7, 2026 | A configurable silence interval for the documented streaming API, not total agent latency |
Approximately 75 ms Flash model inference | ElevenLabs latency guidance, accessed September 7, 2026 | Provider-reported model inference only; network and application delays are additional |
These figures describe different boundaries. They cannot be compared as competing end-to-end benchmarks.
What causes latency in voice AI agents?
Latency comes from processing and waiting throughout the response path. The slowest visible component is not always the slowest model.
Turn detection and latency in speech to text
Voice activity detection, or VAD, identifies speech activity. Turn detection decides whether the user has finished. A pause can mean “your turn,” but it can also mean “I am remembering the rest of my booking number.”
Streaming STT produces text while the caller speaks. That does not mean every partial result is safe to act on. In Deepgram’s documented Nova streaming flow, interim results can change; is_final marks a finalized segment, while speech_final identifies an endpoint. One completed turn may contain several finalized segments.
Inspect whether the STT provider waits for silence and the orchestration layer then adds another wait. LiveKit documents that, in STT mode, its endpointing delay can be applied after the provider’s end-of-speech signal.
Model startup and response generation
Model selection, prompt size, output length, and serving conditions affect response time. Start by measuring TTFT and the time until the first speakable phrase, rather than assuming total token throughput explains the pause.
Anthropic recommends choosing a model suited to the task, reducing unnecessary input and output, and streaming responses. Establish acceptable answer quality before applying those optimizations.
Retrieval and business tools
A fast language model cannot remove a slow reservation API. Retrieval, database calls, authentication, retries, and subsequent model passes can become the longest chain of dependent work.
For example, a booking confirmation may require checking availability, creating the reservation, validating the result, and composing the reply. Record each dependency. A fast “Let me check” is an acknowledgment, not a completed booking.
Speech synthesis and playback
The TTS service may wait for enough input text before generating speech. The application may then wait for a larger audio buffer before playback. Both waits can delay a response even when synthesis itself is fast.
ElevenLabs documents streaming audio output and bidirectional text input for TTS, including the tradeoff between text chunk size and natural speech. Streaming helps users hear the first part while later audio is still being generated.
Media routing, buffers, and load
Geographic distance, relays, unnecessary audio conversions, congested connections, and worker queues can add delay. Under load, a system may spend more time waiting for resources than processing a request.
For browser calls, WebRTC statistics can expose round-trip time, packet loss, jitter, and jitter-buffer measurements. These help diagnose the media path, but none independently measures the full AI response gap.
How to build a voice agent latency budget
A voice agent latency budget allocates the allowed response time across the work that must finish before the caller can hear the answer.
For a deliberately serial model, use:
Response time ≈ final-audio transport + turn/STT readiness + answer preparation + speech startup + return/playback.
In a streaming implementation, stages can overlap. Measure the critical path, meaning the longest chain of dependent work, rather than adding every recorded duration. If STT finalization is already included in a turn-readiness measurement, do not add it again.
Illustrative 800 ms budget
The table below is an original planning example for a warm, no-tool response. It is not a benchmark, provider guarantee, or RTC LEAGUE test result. Every row represents a successive, non-overlapping interval.
Interval | Example allocation | Cumulative time |
Caller’s last speech sample → that sample reaches agent ingress | 50 ms | 50 ms |
Ingress → turn accepted and required transcript ready | 250 ms | 300 ms |
Input ready → first speakable response phrase ready, including request startup | 250 ms | 550 ms |
Phrase ready → first audio chunk received by the agent application | 150 ms | 700 ms |
Chunk received → first audio plays at the caller endpoint | 100 ms | 800 ms |
Total | 800 ms | 800 ms |
If this exact serial example adds a required 600 ms lookup before answer preparation, the total becomes 1,400 ms, assuming all other intervals stay unchanged. Further model passes, retries, or validation would add more time.
These allocations are not per-stage p95 targets. Adding component p95 values does not produce the p95 of the complete response. Calculate the end-to-end percentile from complete, correlated turn measurements.
How do you reduce voice agent latency?
Voice agent latency optimization works best as a repeated cycle: measure a slow turn, identify its blocking interval, make one change, and validate speed and correctness together.
1. Instrument the response path before replacing components
Assign a session ID and turn ID. Capture speech end, turn acceptance, transcript readiness, model requests, first speakable phrase, tool results, first audio, and playback start where observable.
Use monotonic clocks for local durations. Synchronize or calibrate measurements across machines. If a telephony integration only exposes gateway playback, state that boundary and validate handset timing with a controlled call test.
Keep interrupted, unanswered, failed, and timed-out turns in separate counters. Excluding them silently can make a broken agent appear faster. RTC LEAGUE’s AI voice agent monitoring guide provides broader context for connecting technical metrics with call outcomes.
2. Tune endpointing against real speech
Test pauses, self-corrections, digit sequences, accents, background noise, and supported languages. Adjust the silence threshold or turn detector only after listening to the cases where it responds too early or too late.
Start with one clearly understood end-of-turn policy. Check each layer’s configuration for stacked waits. The fastest silence threshold is rarely a useful default for every caller.
3. Stream the pipeline and control speculation
Process incoming audio continuously, make transcript updates available promptly, and stream speakable text into TTS. Avoid waiting for the entire answer when the first phrase can safely be delivered.
Preemptive generation can begin before a turn is confirmed. LiveKit documents this speculative approach and the possibility of discarded work. Use it only with cancellation and transcript-revision handling.
Do not commit purchases, bookings, or other consequential actions from a provisional transcript. After cancellation, prevent stale speech or results from entering the next turn.
4. Reduce LLM latency without removing necessary context
Compare candidate models on the same task set. Remove duplicated instructions, irrelevant history, and unnecessary retrieved material. Request concise spoken responses and route simple tasks to an appropriately capable model.
Shorter output mainly reduces completion time; it does not guarantee a proportional reduction in TTFT. Measure both. Avoid hard token caps that truncate required information.
Where supported, reuse stable prompt prefixes with prompt caching and inspect cache hits. This reduces repeated prompt processing, not the time needed for live business data.
5. Shorten the tool path
Run independent read operations concurrently where the application permits it. Keep dependent operations ordered. Set explicit timeouts, bound retries, and cache only data whose freshness and access rules allow reuse.
When a lookup takes time, offer a truthful acknowledgment such as “I’m checking the available times.” Then deliver the verified result. Track both events so acknowledgment audio does not hide slow task completion.
6. Start speech earlier and keep audio flowing
Choose a streaming TTS interface appropriate to whether text is available upfront or arrives incrementally. Test the first phrase for pronunciation, pauses, and continuity.
If audio stalls while text is available, inspect the generation trigger and text buffer. ElevenLabs warns that a chunk schedule can wait for additional characters and documents flushing remaining text at the end of a turn.
Reduce playback buffering carefully. A smaller buffer must still tolerate the network variation expected in production.
7. Shorten the media path and prepare capacity
Map the locations of the caller, media server, agent worker, speech services, and tools. Test regions using actual traffic patterns and data-location requirements. Place tightly coupled services close together when feasible.
Avoid redundant resampling and transcoding, but preserve the format required by each service. Keep connection setup and worker initialization out of recurring turns where possible. Load-test concurrency, queue growth, reconnects, and provider limits before accepting a result from a quiet development environment.
For implementation support, explore RTC LEAGUE’s LiveKit integration services and WebRTC development services.
8. Make interruption handling part of the latency test
Measure from the caller’s interruption to the moment agent audio stops. Cancel obsolete generation and clear queued playback, then reconcile conversation state with what was actually heard.
Test short backchannels such as “mm-hmm” as well as direct interruptions such as “No, the other date.” An agent that responds quickly but keeps talking over corrections still needs work.
Do speech-to-speech models remove the latency problem?
Native audio models can simplify the explicit STT → LLM → TTS pipeline, but they still have turn detection, model processing, tool dependencies, transport, and playback to manage. Google’s Live API documents streamed audio and configurable voice activity detection, illustrating this architecture.
Architecture | Useful when | What to evaluate |
Cascaded STT → LLM → TTS | You need independently selectable speech and language components | Handoff delays, transcript accuracy, voice quality, and per-stage telemetry |
Native speech-to-speech | You want an integrated audio interaction model | Endpoint response time, tool behavior, interruption control, language support, and observability |
There is no universal winner. Compare both on the same caller scenarios, regions, concurrency, and correctness requirements. Missing STT or TTS spans should be treated as a measurement difference, not as proof that those operations take zero time.
Troubleshooting slow voice agents
Symptom | Inspect first | Next experiment |
Transcript appears quickly, but the agent waits | Turn acceptance and duplicate endpointing waits | Tune one end-of-turn policy using hesitation-heavy recordings |
Model output arrives, but speech does not start | First speakable phrase, TTS trigger, text buffer | Adjust phrase batching and end-of-turn flushing |
Audio reaches the client quickly, but plays late | Decode, scheduling, and playback buffer | Compare receive timestamps with playback events |
Agent is fast locally and slow elsewhere | Service regions, route, RTT, jitter, and relays | Replay equivalent calls from affected regions |
Only lookup turns are slow | Tool spans, retries, and follow-up model calls | Shorten the dependency chain and test a bounded timeout |
Performance degrades at peak traffic | Queues, worker readiness, and provider concurrency | Repeat the test at intended call volume |
Agent interrupts names or numbers | End-of-turn policy and transcript revisions | Use labeled pause and correction cases before reducing waits |
Agent continues after “stop” | Cancellation propagation and queued output | Trace interruption to audible stop at the endpoint |
These are diagnostic hypotheses. Use the trace to confirm the cause before changing the stack.
A validation checklist for low-latency voice agents
Use this checklist to evaluate any implementation or provider:
The latency claim names its start event, stop event, and observation point.
End-to-end p50 and p95 include a sample count and defined test conditions.
Cold starts, peak load, different regions, languages, and phone/browser channels are represented.
Acknowledgment time and verified-answer time are reported separately.
Failed, timed-out, and interrupted turns are accounted for.
Speed improvements preserve critical entity accuracy and task success.
Barge-in reliably stops queued audio and prevents stale responses.
The comparison records model, prompt, voice, region, and application versions.
Recorded calls and transcripts follow the deployment’s consent, access, and retention controls.
Change one major variable at a time. Replay the same evaluation set, compare results by segment, and investigate any quality regression before rollout. Continue checking production traces after deployment because real traffic can reveal waits that a controlled test misses.
How RTC LEAGUE can help
RTC LEAGUE provides AI voice platforms, LiveKit integration, and WebRTC development. That combination is relevant when latency spans the voice pipeline and the infrastructure carrying the call.
In its published LiveKit optimization case study, RTC LEAGUE reports a 40% latency reduction. The public article does not provide enough baseline timing, percentile, or test-condition detail to use that figure as a reproducible benchmark. Treat it as a reported project outcome, not a forecast for another deployment.
Find the delay in your voice agent. Bring a sample call trace, your current response-time distribution, target regions, and expected concurrency. Discuss where time is being spent and which changes to test first.
-(1).jpg)
.jpg)