INQUIRING LINE
Inquiring lines›Where does language-model reasonin…›How do language models represent m…›What articulatory information do s…›this inquiring line

Cutting transcription from voice AI saves more than a step — it lets the model start responding before you finish speaking.

Does direct speech-to-speech generation really eliminate transcription latency?

This explores whether generating speech directly from speech input actually removes the delay that comes from transcribing audio to text first — and what the corpus says is really being saved (and what's being traded away).

This explores whether skipping the transcription step in voice systems genuinely eliminates latency, or just relocates it. The short answer from the corpus is: yes, it removes a real and specific cost — and the reason is more interesting than "fewer steps." The classic voice pipeline is a cascade: speech goes to a recognizer, the recognizer emits text, a language model reads the text, and only then does generation begin. Each stage has to wait for the previous one to finish a unit before it can act. The direct approach in Can skipping transcription make voice assistants faster? collapses that into ~226ms not merely by deleting a box, but because speech embeddings carry acoustic information the model can start acting on before the full utterance arrives — text is a lossy, late-arriving intermediate, and removing it lets generation begin earlier.

But "eliminate" is too clean a word. What the direct model really does is trade one cost for another. The transcription step wasn't only slow — it was also a place where errors got corrected and intent got resolved. Real-world recognizers run at 15–30% error rates in noisy settings, which is exactly why traditional dialogue systems leaned on probabilistic belief-tracking rather than trusting a single transcript (Why do dialogue systems need probabilistic reasoning?). Skip transcription and you also skip that explicit error-handling layer; the burden of coping with ambiguous, messy audio moves inside the model instead of disappearing.

The deeper reframing is that the win isn't "no transcription" — it's "no hand-offs." The most aggressive version of this idea (Can a single model learn when to speak and respond?) treats language, audio, and video as one interleaved token stream so that even turn-taking — knowing when to speak — becomes learned behavior inside a single model rather than a separately engineered module. The latency savings there come from the same source: every boundary between specialized components is a place where one stage waits on another, and unifying them removes the waiting, not just the transcribing.

There's also a reason this works acoustically and not just architecturally. Self-supervised speech models appear to learn the language-agnostic physics of how a vocal tract produces sound rather than discrete phonetic categories (Do speech models learn language-specific sounds or universal physics?). That continuous, sub-symbolic representation is what makes it viable to generate from speech directly — there's rich structure to act on without first quantizing everything into words.

So the honest answer: direct speech-to-speech does eliminate transcription latency specifically, and the gain is real because text was both a delay and a loss of information. But it doesn't eliminate the *work* transcription was quietly doing — error correction and intent disambiguation — it just internalizes it. The latency you save is the cost of waiting for hand-offs between modules; the risk you inherit is that the explicit safety net for noisy, misheard speech is gone, and the model now has to absorb that uncertainty on its own.

Related lines of inquiry

Explore this area

Sources 4 notes

Can skipping transcription make voice assistants faster?

LLaMA-Omni generates speech responses directly from speech input without transcribing to text first, achieving 226ms latency. This works because speech embeddings preserve acoustic information that text loses, enabling generation before full input is received.

Why do dialogue systems need probabilistic reasoning?

Real-world speech recognition achieves 15-30 percent error rates in noisy environments, making deterministic flowchart dialogue systems unworkable. POMDP-based systems handle this by maintaining belief distributions over user intent rather than committing to single interpretations.

Can a single model learn when to speak and respond?

Wan-Streamer represents language, audio, and video as one interleaved causal token stream, allowing response timing and turn management to be learned jointly within a single Transformer rather than engineered as separate modules, achieving sub-second latency.

Do speech models learn language-specific sounds or universal physics?

Self-supervised speech models learn the language-agnostic physics of how the vocal tract produces acoustics, not language-specific phonetic categories. This explains their multilingual transfer and predicts their downstream task performance better than phonetic probing.

Papers this line draws on 8

The research behind the notes this line reads — ranked by how closely each paper relates.

Research prompt for your LLMexpand ↓

Copy into ChatGPT or Claude to take this line of inquiry further — it asks the model to find newer work and re-test which earlier constraints still hold.

You are a speech-systems analyst. Keep this question open: does direct speech-to-speech generation truly eliminate transcription latency, or does it relocate the work transcription was quietly doing?

What a curated library found — and when (dated claims, not current truth) — findings span roughly 2019–2026:
- Deleting the recognizer→text→LM cascade reaches ~226ms response latency, because speech embeddings carry acoustic information the model can act on before the full utterance arrives — text is a lossy, late-arriving intermediate (~2024).
- Recognizers run at 15–30% error rates in noisy settings, which is exactly why traditional dialogue systems leaned on probabilistic belief-tracking instead of trusting a single transcript.
- Skipping transcription also removes the explicit error-correction and intent-disambiguation layer; that burden moves inside the model rather than disappearing.
- Full-duplex designs treat language, audio, and video as one interleaved token stream, so even turn-taking becomes learned behavior inside one model (~2026).
- Self-supervised speech models appear to learn language-agnostic articulatory physics rather than discrete phonemes (~2023), giving continuous, sub-symbolic structure to generate from directly.

Anchor papers (verify; mind their dates): Self-Supervised Models of Speech Infer Universal Articulatory Kinematics (2023); LLaMA-Omni (2024); Wan-Streamer v0.1 (2026); Neural Assistant (2019).

Your task: (1) Re-test each constraint: for every finding, judge whether newer models, training, tooling, orchestration (streaming, caching, multi-agent), or evaluation has relaxed or overturned it — and reconcile the two threads, separating the durable question (where does latency actually live) from the perishable limit (the ~226ms number, the 15–30% error floor, the missing error-handling layer); cite what resolved each, and say plainly where a constraint still holds. (2) Surface the strongest superseding work from the last ~6 months. (3) Propose 2 research questions that assume the regime has moved. Cite arXiv IDs; flag anything you cannot ground in a real paper.