Building Voice Agents for Production: Why Latency, Not Accuracy, Kills You First

Been shipping voice agents for the past few months and realized something: we obsess over transcription accuracy and semantic understanding, but what actually breaks in prod is the latency trap.

Here's what happened. We built a voice agent that performed beautifully in evals - 94% intent recognition, tight context windows, solid error handling. In testing it looked great. Then we hit real users. Turns out a 1.2 second speech-to-text lag kills the user experience far before accuracy matters. People start talking over the agent, the system gets confused, and the whole interaction collapses.

The real operational challenge is managing the chain: audio capture -> STT -> LLM inference -> TTS generation. Each step has latency budgets that cascade. We were budgeting 300ms per component thinking we'd come in under 1.5s total. In practice, with any meaningful semantic complexity, you're looking at 400-600ms for the LLM step alone, not counting network jitter.

What we had to do:

Stream STT output as it arrives instead of waiting for the full transcription. This means your agent is partially responding before the user finishes speaking - which feels natural or horrifying depending on implementation. We added confidence thresholds to avoid responding to incomplete thoughts.

Cache everything aggressively. Intent patterns, entity extraction, knowledge base lookups - all get cached. First request eats the latency, subsequent similar queries snap back instantly. This made a huge difference for repeated user patterns.

Decouple TTS from the critical path. Generate it in parallel while the LLM is thinking, not after. If it takes 1.5s to generate speech but the user waits anyway, that's acceptable if they're not blocked watching it happen.

Use token budgeting like you would in multi-turn text conversations. Voice agents tend to get more verbose - avoid the temptation to let them ramble. Hard limits on generated tokens, even if it means cutting mid-sentence sometimes (better 2 second response than 6 second response).

The evaluation piece everyone's talking about lately actually nails something important - you can't just benchmark accuracy metrics. You need to test latency profiles under load, cache hit rates, what happens when one component degrades. We started running simulations where STT jitter increases 200ms and seeing how the whole chain responds.

Curious if anyone else has hit the latency wall with voice agents. Does your team measure end-to-end latency perception differently, or are you also finding this is the real bottleneck? And how do you handle cases where your inference step just takes time - do you accept degraded UX or architect around it?