Skip the transcript step and an AI voice response drops to 226 milliseconds — fast enough to finally feel like a real conversation.
How much latency improvement comes from collapsing the speech pipeline?
This explores what you actually gain in speed when you stop chaining together separate speech-to-text, language, and text-to-speech stages and instead let one model handle voice end to end.
This explores what you actually gain in speed when you stop chaining together separate speech-to-text, language, and text-to-speech stages and instead let one model handle voice end to end. The corpus gives a concrete number: removing the transcription step entirely lets a system respond in about 226 milliseconds — fast enough to feel like a real conversation rather than a walkie-talkie exchange Can skipping transcription make voice assistants faster?. The reason isn't just that you've deleted a box from the diagram. Speech embeddings carry acoustic information that text throws away, so the model can begin forming a response before the full input has even finished arriving, instead of waiting for a clean transcript to be handed off.
That points to the deeper insight: the latency win comes less from "fewer steps" and more from streaming. A unified model that represents speech, audio, and even video as one interleaved token stream can learn *when* to speak and when to listen as emergent behavior, hitting sub-second response times without bolting on a separate turn-taking module Can a single model learn when to speak and respond?. In a traditional cascade, each module must finish and pass a discrete result downstream — those handoffs are where the dead air lives.
The reason cascades existed in the first place is worth knowing, because it explains what you're trading away. Real-world speech recognition has 15–30% error rates in noisy conditions, and dialogue systems were built around probabilistic reasoning precisely to survive those errors — maintaining a distribution over what the user might have meant rather than committing to one transcript Why do dialogue systems need probabilistic reasoning?. Collapsing the pipeline removes the explicit transcript that error-handling stage operated on, so the speed gain comes with a quieter bet: that the unified model absorbs that robustness internally rather than discarding it.
Worth a sideways glance: latency in voice systems isn't only an architecture problem, it's also a decoding problem. Diffusion language models attack the same wall from a different angle, recovering speed by generating blocks of tokens in parallel rather than strictly one at a time Can diffusion language models match autoregressive inference speed?. So the headline answer — roughly a fivefold-plus drop into the low-hundreds-of-milliseconds range — is what pipeline collapse buys, but the corpus suggests the real lever is letting generation start early and run continuously, whichever method gets you there.
Sources 4 notes
LLaMA-Omni generates speech responses directly from speech input without transcribing to text first, achieving 226ms latency. This works because speech embeddings preserve acoustic information that text loses, enabling generation before full input is received.
Wan-Streamer represents language, audio, and video as one interleaved causal token stream, allowing response timing and turn management to be learned jointly within a single Transformer rather than engineered as separate modules, achieving sub-second latency.
Real-world speech recognition achieves 15-30 percent error rates in noisy environments, making deterministic flowchart dialogue systems unworkable. POMDP-based systems handle this by maintaining belief distributions over user intent rather than committing to single interpretations.
Discrete Diffusion Forcing breaks the speed barrier through block-wise autoregressive generation with KV cache reuse and inter-block parallel decoding. This hybrid approach recovers both the compute efficiency of AR and the parallelism advantage of diffusion.
Papers this line draws on 8
The research behind the notes this line reads — ranked by how closely each paper relates.
- Intent Mismatch Causes LLMs to Get Lost in Multi-Turn Conversation
- Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
- Efficient Streaming Language Models with Attention Sinks
- Diffusion LLMs Can Do Faster-Than-AR Inference via Discrete Diffusion Forcing
- Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models
- POMDP-based Statistical Spoken Dialogue Systems: a Review
- Large Language Diffusion Models
- A Survey on Diffusion Language Models