Speech-to-Speech

Speech-to-speech AI takes audio in and returns audio out — conversation or translation — via a cascade (ASR to text to TTS) or one end-to-end model.

Published Updated

On this page

Definition

Speech-to-speech is the task of taking spoken audio in and producing spoken audio out with an AI system in between — you talk, and it talks back. The reply can be in the same language, as with a voice assistant that answers a question aloud, or in a different language, as with live spoken translation. What makes the term worth its own page is not the input and output, which are obvious, but the choice hidden between them: whether the system passes your words through text on the way, or never writes them down at all. That single architectural decision sets how fast it responds and how much of how you spoke survives the trip.

Two systems can both listen and reply in a natural voice and yet be built in completely different ways. The older way, a cascade, transcribes your speech to text, runs the text through a language or translation model, and reads the answer back with a separate voice. The newer way, an end-to-end or native-audio model, treats audio as its native currency from start to finish. The cascade is a relay race with three runners; the end-to-end model is one runner who covers the whole distance.

How It Works

The cascade: three models in a row

A cascade wires together three components that already exist and are individually mature:

  • Speech recognition (ASR) turns your audio into a transcript. See Voice Recognition for how this stage works.
  • A language or translation model reads that transcript and decides what to say back — answering a question, or rendering the sentence in another language.
  • Text-to-speech (TTS) turns the reply into audio. See Text-to-Speech (TTS).

This design dominated for years because each piece could be trained on abundant, well-understood data and swapped out independently, and because the intermediate transcript is something engineers can log, read and correct. Its weakness is structural: the stages run in series, and text is a lossy interface between them.

Why the cascade is slow: latency is a sum

Because the three stages run one after another, their delays add up. Suppose a reader asks a spoken question and we use figures typical of production voice systems as of 2026:

Stage                                        Delay
-----------------------------------------    -------
ASR: wait for the speaker to stop, then
     finish transcribing (endpointing +
     recognition)                            ~300 ms
Language/translation model: read the
     transcript, produce the first token     ~400 ms
TTS: turn the first sentence into audio      ~200 ms
-----------------------------------------    -------
Total before the user hears anything         ~900 ms

That is close to a full second of silence before a single sound comes back, and it is optimistic — it ignores the network hops between three separate services and any queuing. The arithmetic itself is the durable point, not the exact milliseconds:

L_cascade = L_asr + L_llm + L_tts + transport

Latency is additive across serial stages. Speeding up any one stage barely dents a sum of three — halving the 200 ms TTS saves 100 ms out of 900. The delay is also partly baked in: ASR normally waits to be sure you have stopped talking (endpointing) before it commits a transcript, and the language model wants a finished sentence before it reasons well, so the stages mostly block each other rather than overlapping.

The end-to-end model: collapsing the sum

A native-audio model consumes your audio and emits audio from one network, with no transcript in the middle. It does not have to wait for a finished transcription to exist before it starts understanding, and it can begin speaking as soon as it has decided how to respond. That turns the three-term sum into a single time-to-first-audio:

L_end_to_end = L_model    (one term, not three)

Removing two of the three terms — not shrinking them — is the win. As of 2026, OpenAI states that its Realtime API responds in under 500 ms; Google's native-audio Gemini models answer at a similar conversational pace. Those specific figures will move, but the reason a single model can hit them while a cascade cannot is the arithmetic above.

What text throws away

The second cost of routing through text is subtler and often matters more than speed. A transcript records which words were said and nothing about how. The sentence "I'm fine" is identical on the page whether it was said sincerely, wearily or with open sarcasm; pitch, emphasis, pace, hesitation, laughter, a shaky voice and who is speaking all vanish the moment audio becomes text. A cascade's TTS then has to guess an emotional delivery for the reply, because the information it would have needed was discarded two stages earlier.

An end-to-end model can carry that expressive layer straight through. Meta's research makes the mechanism explicit: their SpiRit-LM model (2024) interleaves speech and text tokens in one stream so that a single model shows, in the paper's words, "both the semantic abilities of text models and the expressive abilities of speech models," and its Expressive variant adds pitch and style units on top of the semantic ones precisely to model how something is said, not just what. Keeping audio as the interface is what lets prosody and emotion survive.

Types

Speech-to-speech splits along two independent axes, and it helps to keep them separate.

The architecture axis is the cascade-versus-end-to-end distinction above, and it is a real, named split that practitioners choose between — not an invented taxonomy.

The task axis is what the system is for:

  • Same-language conversation. You speak, it replies in the same language: a voice assistant, a spoken customer-service agent, a talking companion. The hard part is latency and natural turn-taking.
  • Speech-to-speech translation (S2ST). You speak one language and it speaks another. This can itself be a cascade (ASR to machine translation to TTS) or direct. Meta's SeamlessM4T (2023) is a single, unified model that performs direct speech-to-speech translation across around 100 languages, an example of the end-to-end approach applied to translation rather than conversation.

Real-World Applications

Speech-to-speech has moved from demo to product across several named systems, and each is a decision someone now makes differently:

  • Native-audio voice assistants. OpenAI's Realtime API (its gpt-realtime family) and Google's native-audio Gemini models both process audio in and audio out directly rather than orchestrating a transcriber and a voice service; as of 2026 they power spoken assistants that detect emotion and handle interruptions. These are the current frontier of same-language conversational speech-to-speech, and naming the specific model is less useful than knowing the category, because the leader changes.
  • Live spoken translation. Google rolled out live speech-to-speech translation in the Translate app and through headphones (2025–2026), translating between speakers in real time while keeping tone and pacing; the underlying models cover 70-plus languages at the time of writing. This is speech-to-speech translation reaching consumers rather than researchers.
  • Open translation research and tooling. Meta's SeamlessM4T is openly released, so it anchors a great deal of academic and product work on direct S2ST that does not depend on a single vendor's API.
  • Accessibility and call handling. Spoken interfaces let people interact with software hands-free and eyes-free, and voice agents in call centers now answer, understand and reply in speech without a human on the line.

Challenges

The problems that remain are specific to putting audio on both ends of the pipe.

  • Evaluation has no word error rate. There is no single correct output waveform, and prosody, naturalness and emotional appropriateness have no agreed metric. Teams often transcribe the generated speech and score that (ASR-BLEU), but doing so runs the output back through a recognizer and reintroduces exactly the recognition errors the end-to-end design was meant to avoid.
  • Losing the transcript costs you observability. The cascade's intermediate text is a gift for debugging, logging and moderation — content filters historically ran on it. An end-to-end model produces no transcript to inspect, so a mistranslation or an unsafe reply is harder to trace, and safety tooling built for text has to be rebuilt for audio.
  • Parallel speech data barely exists. Training a direct model wants aligned audio pairs — the same content spoken in two languages, or a spoken prompt paired with a spoken reply — and that data is scarce next to the enormous ASR, translation and TTS corpora a cascade reuses. Data scarcity is a large part of why cascades held on so long.
  • Streaming forces early commitment. To answer within a few hundred milliseconds, a model may start speaking before it has heard the whole utterance, risking a confident reply to a sentence that had not finished. Balancing responsiveness against being wrong is an open trade-off.
  • Realistic voices are a safety surface. A model that can carry and generate emotion and timbre is a short step from voice cloning, raising consent and impersonation risks that a robotic cascade never posed.

Future Trends

The clearest near-term direction is full-duplex interaction: systems that listen and speak at the same time, so you can interrupt ("barge in") and the model stops, the way people actually converse, instead of the rigid you-talk-then-I-talk turns of early voice assistants. Alongside that, native-audio models are on track to become the default way software talks with people rather than a specialist option, and pushing them onto devices would cut both the network latency and the privacy exposure of sending a live microphone stream to a server.

Frequently Asked Questions

Text-to-speech (TTS) starts from written text and produces audio. Speech-to-speech starts from spoken audio and produces spoken audio — it has to understand what was said before it can reply or translate. A speech-to-speech system usually contains a TTS stage, or replaces it with a single model that emits audio directly.
A cascade chains three separate models: speech recognition turns audio into text, a language or translation model processes that text, and text-to-speech reads the answer aloud. An end-to-end (native-audio) model does all three jobs at once, mapping audio to audio without a text bottleneck in the middle. The cascade is easier to build and debug; the end-to-end model is faster and keeps tone and emotion the text would have thrown away.
Increasingly yes. Live speech-to-speech translation now runs in consumer apps and headphones, translating a speaker into another language within a second or so while they keep talking. Same-language voice assistants built on native-audio models respond in a few hundred milliseconds, close to the rhythm of human conversation.
Because they never convert the audio to plain text, where that information does not exist. A transcript records the words 'I'm fine' but not whether they were sincere or sarcastic. A model that reads and writes audio can carry pitch, emphasis, pauses and laughter straight through from input to output.
Speech-to-text has word error rate: compare the transcript to a reference and count mistakes. Spoken output has no single correct waveform, and prosody, emotion and naturalness have no agreed metric. A common workaround is to transcribe the generated speech and score that — but doing so reintroduces the recognition errors the system was built to avoid.

Related Terms

Text-to-Speech (TTS)

Text-to-Speech (TTS) is an AI technology that converts written text into natural-sounding human speech. Modern TTS uses deep learning to capture emotion.

Voice Recognition

How AI turns spoken words into text (speech recognition / ASR) — and how that differs from identifying who is speaking (speaker recognition).

Conversational AI

Conversational AI holds a multi-turn dialogue in natural language. Unlike an intent-based chatbot, it generates each reply rather than picking one.

Multimodal AI

How one model reads an image and text at once: the picture is cut into fixed patches, each patch becomes a token, and those tokens cost context.

Audio Processing (AP)

How computers turn sound into numbers: sampling, the Nyquist limit, spectrograms and mel filterbanks — the front end of every speech and audio model.

Continue Learning

Explore our use-case guides and prompts to deepen your AI knowledge.

Browse AI Tools← Back to Glossary