Cutting a second off voice AI response time
Each stage adding a third of a second produces an unusable one-and-a-half second gap.
AI Tools Hub Editorial · published · updated
The stages
| Stage | What happens | How to cut it |
|---|---|---|
| End-of-speech detection | Deciding the caller finished | Shorten the silence threshold, at the risk of interrupting |
| Transcription | Audio to text | Use a streaming engine |
| Inference | Deciding the reply | Faster model, shorter prompt |
| Synthesis | Text to speech | Low-latency engine, sentence-level streaming |
| Network | Round trip | Closer region |
Start with silence detection
A one-second silence threshold puts a floor under everything else. Shortening it causes interruptions, and Japanese places meaning at the end of the sentence, so English defaults interrupt more often. This needs tuning on real calls.
Stream synthesis by sentence
Waiting for full generation before synthesising adds the whole generation time to first audio. Sending each completed sentence to synthesis cuts that dramatically, at the cost of not being able to revise mid-utterance.
Fill what you cannot remove
A brief acknowledgement during processing buys more patience than silence. It is an implementation change, not a model change.
FAQ
What is acceptable?
Above one second on a phone call, people start checking whether the line is live. Target half a second and cover the rest with audio.