When inference speed decides the experience
Latency is not a universal priority. In three specific situations it dominates everything else.
AI Tools Hub Editorial · published · updated
Three cases
Voice: half a second of silence on a phone call reads as the system failing to understand. Transcription, inference and synthesis stack their delays.
Autocomplete: a suggestion that arrives after the developer has typed the line is worse than none, because it interrupts.
Multi-step agents: five steps at two seconds each is ten seconds. Most of those steps do not need a frontier model.
Where the time goes
- Time to first token, which grows with prompt length
- Token generation rate, which matters for long outputs
- Network round trip
- Everything around the model: retrieval, transcription, synthesis
Streaming first
Streaming does not reduce total time but cuts perceived waiting by a large factor, and it is cheap to implement. Voice is the exception: you cannot unsay a streamed sentence, so synthesis needs sentence-level chunking.
Measure before optimising
Teams swap to a faster model and discover retrieval was the slow step. Record per-step timings first, then fix the longest one.
FAQ
Are faster models worse?
Smaller models are generally faster and weaker at complex reasoning, but the gap disappears on classification and extraction. Use different models per step.