Spending a latency budget on voice
A voice agent that thinks for two seconds feels broken even when the answer is perfect. Here is where our 900 milliseconds go, and the three places we found to buy time back.
Text buys you patience. A chat widget can take two seconds to answer and nobody minds, because a typing indicator is a socially complete explanation for a pause. Voice has no equivalent. Silence on a call reads as a problem with the call.
We set a target of 900 milliseconds from the moment a caller stops speaking to the moment they hear the first syllable back, and then went looking for where that time was actually going.
The budget
Roughly, and on a good network: endpointing takes about 150ms, transcription 120ms, the model's first token 380ms, and speech synthesis 180ms before audio starts flowing. That adds to 830ms, which leaves almost nothing for the network or for the workflow itself to do any thinking.
The uncomfortable finding was that the model was not the biggest problem. Endpointing was, because we had tuned it conservatively to avoid interrupting people mid-sentence, and that caution was being paid on every single turn.
Buying time back
- Adaptive endpointing. The wait before we decide someone has finished now depends on what they were saying. Trailing off after a complete question ends the turn faster than trailing off mid-clause.
- Speculative tool calls. When the transcript so far makes a lookup near-certain, we start it before the model asks. If the model asks for something else, we throw the result away, which costs an API call and saves a few hundred milliseconds when we guessed right.
- Streaming into synthesis. We no longer wait for a full sentence. Synthesis starts on the first clause boundary, so the caller hears the beginning of the answer while the rest is still being generated.
Perceived speed is mostly about when sound starts, not when it finishes.
What we did not do
We did not switch to a smaller model. The two attempts we made traded about 200ms for answers that were noticeably worse at following guardrails, and a fast wrong answer on a customer call is worse than a slow right one. That trade may be worth revisiting when the next model generation lands.
Filler is a last resort
Every voice product eventually considers playing "let me check that for you" over a slow lookup. We shipped it, gated behind a threshold, and it only plays when a tool call has already run past 1.2 seconds. Used sparingly it covers a genuine wait. Used on every turn it becomes a verbal tic that callers start finishing for you.