Definition
Voice cloning is the use of AI to reproduce a specific person's voice, so that a synthesizer can make it say words the person never recorded. What makes it a distinct capability — rather than just text-to-speech with a pleasant voice — is how little of the target it needs. A modern zero-shot system can copy an unfamiliar speaker from about three seconds of reference audio, where building a custom synthetic voice a decade ago meant recording that one person for tens of minutes to many hours in a studio.
That collapse in the enrollment cost — from hours down to seconds — is the whole story of the term. It moved voice cloning out of the studio and into reach of anyone with a clip scraped from a podcast, a voicemail or a conference talk. The consequence a reader should leave with is blunt: a voice on the phone can no longer be treated as proof of who is speaking. That single fact is why banks that rolled out "your voice is your password" authentication have been quietly retiring it, and why the same technology sits behind both a screen-reader that sounds like its user and a scam call in a relative's voice.
Voice cloning is one flavour of deepfake — the audio one. The cloned voice is frequently the soundtrack of an audiovisual fake, but the audio capability stands on its own, because it is far cheaper to produce than convincing video and it travels down a channel, the telephone, that carries no picture to contradict it.
How It Works
Voice cloning is built on the same neural audio codec pipeline that modern text-to-speech uses, so it is worth reading the TTS page for how a waveform becomes a short sequence of discrete tokens. The part that is specific to cloning is how the target speaker's identity gets into the output, and there are two routes.
In-context prompting. The breakthrough that made seconds-of-audio cloning ordinary treats the problem as language modelling over audio tokens. Microsoft's VALL-E (2023) encodes the three-second reference clip into codec tokens, prepends them to the phoneme sequence for the text to be spoken, and then predicts the continuation of the token sequence the way a language model predicts the next word — except the "words" are chunks of sound. Because the continuation has to remain consistent with the prompt tokens, it comes out in the same voice, with the same accent and even the same recording conditions. The clip is never used to train or update the model; it is context at inference time, which is why the method is called zero-shot.
The reason three seconds is enough is that the model has already learned what human voices are. VALL-E was trained on 60,000 hours of English speech from over 7,000 speakers — hundreds of times more than earlier TTS systems. The three-second prompt does not teach the model a new voice; it points the model at a region of the voice space it already learned, the way a single example steers a language model that has read the whole internet. This is the same insight that underpins zero-shot TTS, applied to speaker identity.
Speaker embeddings. The older and simpler route runs the reference clip through a dedicated encoder that outputs a fixed-length vector — a speaker embedding — capturing the timbre and pitch that make the voice recognisable. Synthesis is then conditioned on that vector. This is more constrained than in-context prompting: an embedding compresses the voice down to a point, so it captures who is speaking better than how they were speaking on that particular clip, and it typically carries less of the emotion and room acoustics of the sample.
Types
The practical distinction that matters is how much of the target you enroll and whether the model changes, and it maps directly onto the shift described above.
- Zero-shot cloning conditions a pre-trained model on a short reference — roughly three seconds — and updates no weights. It is instant and needs almost no data, which is exactly what makes it both convenient and dangerous, because the reference can be any public recording.
- Fine-tuned (speaker-adapted) cloning records more of the target and adjusts the model to that speaker. Apple's Personal Voice (2023) is the mainstream example: the user reads about 15 minutes of prompts and the phone builds a personal voice on-device. It costs more effort and audio than zero-shot but gives higher fidelity and more control, and — as with Personal Voice — it is usually done to clone the user's own voice with consent.
Real-World Applications
Accessibility and voice banking. This is the use with the clearest human stake. Apple's Personal Voice, introduced in 2023, lets someone record around 15 minutes of audio, processed entirely on-device, to create a synthetic version of their own voice; it is aimed explicitly at people at risk of losing the ability to speak, such as those recently diagnosed with ALS. A person can "bank" their voice while they still have it and keep speaking in it afterwards through a text-to-speech interface.
Dubbing and localisation. Cloning lets a film, video or course be re-voiced in another language while keeping the original performer's voice, rather than replacing it with a different dubbing actor. This is the commercial volume for the technology, and it stacks the same constraints as any dubbing work: the translated line has to carry the speaker's identity, match the emotion and fit the on-screen timing.
Media restoration. Cloning can recreate a voice that illness or age has taken. Val Kilmer, whose speech was affected by throat cancer treatment, had his voice reconstructed from archival recordings for Top Gun: Maverick (2022) — a widely reported case of restoration done with the performer's consent and involvement, which is the line that separates it from the misuse below.
Key Concepts
- Enrollment audio: the reference recording a system needs to establish the target voice. Its length is the single number that most distinguishes the methods — seconds for zero-shot, minutes for fine-tuned.
- Speaker embedding: a fixed-length vector summarising a voice's identity, used to condition synthesis in the older route.
- Acoustic prompt: the reference clip used in context in the VALL-E-style route, where identity is carried by the prompt's tokens rather than a distilled vector.
- Voice conversion: a related but distinct task — transforming one person's speech into another's voice while preserving the original words and timing, rather than generating new speech from text.
Challenges
Consent and impersonation are the defining problem, not a side note. Because a usable clone now takes seconds of found audio, cloning is a fraud primitive. The landmark instance is dated and concrete: two days before the New Hampshire primary on 23 January 2024, robocalls carrying an AI-cloned voice of President Biden told Democratic voters that voting in the primary would keep them from voting in November — a fabrication designed to suppress turnout. The FCC finalised a $6 million forfeiture against the political operative behind the calls, Steve Kramer, and separately reached a $1 million settlement with the carrier that transmitted them, Lingo Telecom (down from a proposed $2 million). The same capability drives smaller-scale fraud — calls impersonating a grandchild in distress or an executive authorising a wire transfer — where there is no regulator and no headline.
Detection is losing to generation. There is no reliable way to look at an audio clip and decide it is synthetic, and the gap widens as models improve. This is why the mitigation being standardised is provenance rather than detection: signing content at the moment it is generated (Content Provenance (C2PA)) or embedding an inaudible marker into synthesized audio (Watermarking (SynthID)), so that authenticity is asserted by the honest producer rather than inferred from the fake. Neither helps against a bad actor running an open model that omits the marker, which is the limit of the approach.
Consent as the control point, and the regulatory response. Because the technology itself cannot be recalled, the enforceable question becomes whether the speaker agreed. The FCC's 2024 actions establish that using a cloned voice to deceive is unlawful under existing telephone rules, and lawmakers have advanced proposals to protect a person's voice likeness directly. Responsible providers gate cloning behind consent checks — verifying that the person enrolling a voice is its owner — but that safeguard only binds providers who choose to be bound.
Future Trends
The boundary this page draws is dissolving into real-time pipelines. Speech-to-speech models take audio in and emit audio out without passing through text, which turns cloning into live voice conversion: a caller's words spoken instantly in a target person's voice. That removes the last friction from telephone impersonation and moves the threat from pre-recorded robocalls to interactive, real-time deception.
Against that, the defensive centre of gravity is shifting from detecting fakes to authenticating originals — provenance and watermarking applied at generation, and consent verification applied at enrollment. Both are governance moves rather than technical ones, because the underlying capability is now cheap, open and impossible to withdraw. The durable question for voice cloning is no longer whether a voice can be copied — it can, from almost nothing — but whether a listener can establish that a given recording is genuine.