Definition
Text-to-Speech (TTS) turns written text into an audio waveform that sounds like a person reading it. It is what lets a screen reader voice a web page, an assistant answer out loud, and an audiobook exist without a narrator in a booth.
The thing worth understanding about modern TTS is what it has to produce. Speech at a 24 kHz sample rate is 24,000 numbers per second of audio, and every one of them has to be consistent with its neighbours or the result is noise. The first neural system to sound convincingly human — WaveNet in 2016 — generated exactly that, one sample at a time, and was far slower than real time as a result. It proved the quality was reachable and was useless as a product.
Nearly everything since has been a way around that number. Today the text is not mapped to samples at all: it is mapped to a sequence of discrete audio tokens produced by a learned codec, on the order of 50 to 100 tokens per second, and a separate decoder expands those tokens back into a waveform. Predicting 75 things per second instead of 24,000 is a reduction of more than 300×, and it is what turned speech synthesis into a problem a transformer could handle with the same machinery used for text.
The path from WaveNet to now ran through a few dated steps, and knowing them explains what current systems are. Google's Tacotron 2 (2017) mapped text to a mel-spectrogram with a sequence-to-sequence network and kept WaveNet only as the vocoder, which made end-to-end neural TTS practical. Microsoft's VALL-E (2023) dropped the spectrogram entirely: it treated the codec tokens as a language and predicted them with a transformer trained on 60,000 hours of speech, which let it copy an unseen speaker from a three-second sample. A parallel branch replaced autoregressive token prediction with flow matching — a system such as F5-TTS (2024) denoises the whole utterance in a few steps rather than emitting one token at a time. The names a reader actually meets sit on these mechanisms: ElevenLabs is a voice-AI company (founded 2022), not a model, and the TTS voices in OpenAI's and the major clouds' APIs are hosted products; MOSS-TTS (2026) is an open, downloadable model of the same autoregressive-codec kind.
How It Works
A modern system runs three stages, and it is worth knowing which one fails when something sounds wrong.
Text normalisation and analysis. Before any audio exists, the text has to be resolved into words. "Dr. King Jr. Blvd" contains an abbreviation that is doctor in one context and drive in another; "1/2" is a half or the first of February; "read" and "lead" and "bass" each have two pronunciations that only meaning distinguishes. This stage also decides prosody — where the emphasis falls, where the sentence rises. Most embarrassing TTS errors are born here, not in the neural network everyone credits.
Acoustic modelling. A neural network maps the analysed text to an intermediate acoustic representation — historically a mel-spectrogram, now more often a sequence of codec tokens. This is where speaker identity, emotion and pacing are decided. A voice-cloned model conditions this stage on an embedding extracted from a reference recording, which is why a few seconds of source audio is enough: the model is not learning the voice, it is being pointed at a region of a voice space it already learned from thousands of speakers.
Vocoding. A decoder turns the intermediate representation into the actual waveform. Because this stage is the one that has to emit tens of thousands of samples per second, it is where the engineering effort goes, and it is why streaming systems emit audio in small chunks rather than rendering the whole utterance and then playing it.
Real-World Applications
Accessibility, which came first and still matters most. Screen readers — VoiceOver, NVDA, JAWS — are the reason TTS existed for decades before it sounded good, and their users are the ones who most feel the difference between the old concatenative voices and current models. Experienced screen-reader users often run synthesis at three or four times normal speed, a requirement that trades directly against the expressive prosody newer models are marketed on.
Conversational assistants. Voice modes in chat assistants, in-car systems and phone agents all live under the latency constraint: people leave roughly 200 milliseconds between turns in conversation, so what matters is time-to-first-audio, not total render time. A system that produces a beautiful clip one second late feels broken; one that starts speaking immediately and streams the rest does not.
Media production and localisation. Audiobook narration, video voiceover and dubbing are the commercial volume. Dubbing is the interesting case because it stacks constraints: the translated line must carry the original speaker's voice, match the emotion, and fit the time the actor's mouth was moving.
Call centres and IVR. Replacing pre-recorded prompt trees with synthesis is unglamorous and enormous — it means a system can say an account balance, a date or a name it has never been recorded saying.
Games. Non-player dialogue is the case where the economics are clearest: a studio cannot book actors for every branch of a dynamic conversation, so lines that would never have been recorded at all get spoken.
Key Concepts
- Prosody: the rhythm, stress and intonation carried on top of the words. It is what separates a sentence read correctly from a sentence read meaningfully, and it is what current models improved most.
- Zero-shot voice cloning: conditioning synthesis on a short reference clip rather than fine-tuning a model per speaker. Seconds of audio, not hours.
- Neural audio codec: the learned compression that turns a waveform into a short sequence of discrete tokens and back. The reason TTS became a sequence-modelling problem.
- Streaming synthesis: emitting audio in chunks as the text is processed, so playback starts before generation finishes.
Challenges
Pronunciation of things that are not words. Proper nouns, drug names, ticker symbols, code identifiers and any language mixed into another sentence remain unreliable, because they are exactly the cases with no training precedent and no pronunciation rule. Production systems ship lexicon overrides for this and always have.
Consent and impersonation. Cloning a voice from a few seconds of public audio is now easy enough that voice is no longer usable as proof of identity. In January 2024 an AI-cloned voice of President Biden was robocalled to New Hampshire primary voters telling them not to vote; the FCC proposed a $6 million fine — the agency said it was its first penalty involving generative AI. The same capability drives fraud calls impersonating relatives and executives, and it is why banks that adopted voice authentication have been quietly retiring it. See deepfake for the broader pattern and content provenance for the mitigation being standardised.
The latency–quality trade. The best-sounding models are the largest and slowest. Every real deployment picks a point on that curve, and conversational products pick differently from audiobook pipelines.
Evaluation. There is no loss function for "sounds natural". The field still relies on human mean opinion scores, which are expensive, slow, and not comparable between studies — so claimed improvements are hard to verify.
Future Trends
The direction of travel is away from TTS as a separate component. Speech-to-speech models take audio in and emit audio out without ever passing through written text, which removes the normalisation stage and its errors, preserves the tone of the incoming speech, and cuts a round trip out of the latency budget. That also dissolves the boundary this page describes: when one multimodal model hears and speaks, "text-to-speech" stops being a stage in a pipeline and becomes one capability among several.