Voice Agent API and Latency: Why the LLM Is the Bottleneck (2026)

The voice pipeline

Nearly every voice agent, whether it runs on a managed platform or a self-hosted framework, is the same four stages stitched together:

1. Speech-to-text (STT). Streams the caller's audio into text, with endpointing to detect when they stop talking. Deepgram, AssemblyAI, and Cartesia are common.
2. LLM. Takes the transcript plus conversation history and system prompt, and streams back a reply. This is the reasoning stage and the slowest one.
3. Text-to-speech (TTS). Turns the reply text back into audio, ideally starting to speak the first sentence while later tokens are still arriving. ElevenLabs and Cartesia are common.
4. Telephony and transport. Carries the audio over SIP or a carrier like Twilio, and handles interruptions when the caller talks over the agent.

A managed platform (Vapi, Retell, Bland, Synthflow, ElevenLabs Agents) runs all four for you; a framework like LiveKit or Pipecat lets you wire them yourself and self-host. Either way, the stages are the same, and the LLM is the one you have the most control over and the most to gain from.

The latency budget

"Voice-to-voice" latency is the gap from the caller finishing their sentence to the agent starting to speak. Here is where the ~800ms goes in a typical stitched pipeline:

Add the low ends and you are at ~590ms; add the high ends and you blow past 1700ms. The stages that stay put are STT and TTS. The one that moves the total from "natural" to "broken" is the LLM.

Why the LLM is the lever

Two reasons the model dominates. First, size: at 350 to 1000ms, the LLM's time-to-first-token is the biggest single line in the budget, larger than STT and TTS combined. Second, variance: STT and TTS are roughly fixed across vendors, but the LLM's time-to-first-token swings by a factor of three or more depending on the model's size, the context length, and whether it is on a shared or a dedicated endpoint.

A practical design target cited across voice-infra writeups is an LLM time-to-first-token around 500ms. Hit that and the full pipeline lands inside the ~800ms ceiling. Miss it, at 1000ms or more, and no amount of fast STT or TTS saves the call. There is a second, quieter fact: speech itself is only about 5 to 8 tokens per second of actual spoken words, so once the model starts streaming, it does not need enormous throughput to stay ahead of the voice. Time-to-first-token, not tokens-per-second, is what the caller feels.

Choosing a model for voice

The priorities for a voice model are different from a coding or research model. In order:

• Low time-to-first-token. The one metric the caller feels. Target under ~500ms, and prefer a dedicated endpoint so it stays there.
• Reliable streaming. The model must stream tokens so TTS can start speaking the first sentence while the rest arrives. A model that returns the whole reply at once cannot pipeline.
• Enough, not maximal, throughput. Speech consumes 5 to 8 tokens per second, so anything above ~40 tokens per second comfortably stays ahead of the voice. Throughput past that does not shorten the perceived pause.
• Right-sized for the task. A voice agent handling scheduling, triage, or FAQs does not need a frontier model. A smaller dense model answers faster and costs less per minute.
• Context for the call. Long calls, retrieved documents, and tool outputs add up; a large context window avoids truncating conversation history mid-call.

This is why a smaller dense model on a dedicated endpoint usually beats a large model on a shared endpoint for voice: it starts talking sooner, holds a tighter latency distribution, and costs less per minute, and the caller never needed the extra capability.

Wiring a custom LLM into a voice platform

Any platform that supports bring-your-own-LLM takes an OpenAI-compatible chat endpoint. The mechanism differs, the requirement does not: a streaming /chat/completions route.

{
  "model": {
    "provider": "custom-llm",
    "url": "https://api.morphllm.com/v1",
    "model": "morph-gemma4-31b"
  }
}

Vapi calls {url}/chat/completions and streams the reply, so any endpoint that speaks the OpenAI protocol works unchanged. Retell is different: you host a WebSocket server that speaks Retell's JSON protocol and calls your model behind it. ElevenLabs Agents connects a custom model through a server integration. On a self-hosted LiveKit or Pipecat stack you configure the LLM node directly with a base URL and model name.

You can confirm the endpoint works with a plain OpenAI client before wiring it into the platform:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.morphllm.com/v1",
    api_key="sk-...",  # your Morph API key
)

stream = client.chat.completions.create(
    model="morph-gemma4-31b",
    messages=[{"role": "user", "content": "Thanks for calling, how can I help?"}],
    stream=True,
)
for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="")

Gemma 4 for voice on Morph

Morph serves Gemma 4 31B (morph-gemma4-31b) behind one OpenAI-compatible API at api.morphllm.com/v1, which is exactly the shape every bring-your-own-LLM voice platform expects. It fits the voice priorities above: a dense 31B model is small enough to answer fast, it is multimodal, and it has a 175k context window for long calls and retrieved context.

For voice specifically, the recommended setup is a dedicated single-tenant endpoint. The shared API is fine for testing, but on a phone call the tail of the latency distribution is dead air the caller hears. A dedicated endpoint removes the multi-tenant queueing and delivers sub-300ms time-to-first-token, which leaves comfortable room inside the ~800ms budget for STT, TTS, and the network. Point Vapi, Retell, ElevenLabs, or your own LiveKit stack at it by base URL and you own the latency, the cost per minute, and where your call data lives.

The model and infra layer

Underneath every voice platform sits the same set of component providers. The platform picks defaults; on a bring-your-own stack you pick each one. This is the layer that decides latency and cost.

STT and TTS are close to solved and roughly fixed. The orchestration layer is a build-vs-buy choice. The LLM is the one component that is both the largest latency contributor and fully under your control on any bring-your-own-LLM platform, which is why it is where the latency work pays off.

Frequently asked questions

What latency should a voice agent have?

Under ~500ms feels natural, up to ~800ms is acceptable, past ~1000ms feels broken. The budget: STT 100 to 300ms, LLM 350 to 1000ms, TTS 90 to 200ms, network 50 to 200ms. Optimize the LLM's time-to-first-token first, it is the biggest and most variable piece.

Why is the LLM the bottleneck in a voice agent?

STT and TTS are roughly fixed at 100 to 300ms and 90 to 200ms across vendors. The LLM's time-to-first-token ranges from 350 to over 1000ms depending on model size, context length, and whether it is on a shared or dedicated endpoint. It is both the largest component and the one that varies most, so it dominates the total.

How do you use a custom LLM in a voice agent?

On Vapi, set the model provider to custom-llm with any OpenAI-compatible base URL and model name. On Retell, host a WebSocket server that calls your model. ElevenLabs connects a custom model via a server integration. On self-hosted LiveKit or Pipecat, configure the LLM node directly. In every case the model needs a streaming OpenAI-compatible chat endpoint.

What is the best LLM for a voice agent?

One with a low time-to-first-token, reliable streaming, and enough throughput to stay ahead of speech (5 to 8 tokens per second). A smaller dense model on a dedicated endpoint usually wins on latency. Morph serves Gemma 4 31B (a dense 31B multimodal model, 175k context) and recommends a dedicated single-tenant endpoint for voice, which delivers sub-300ms time-to-first-token.