Voice Recognition

Definition

Voice recognition is the technology that turns spoken language into text — you talk, and a computer writes down the words. This is what powers dictation, live captions, meeting transcripts and the part of a voice assistant that figures out what you said. Its technical name is automatic speech recognition (ASR), or speech-to-text, and answering what was said is what most people searching for "voice recognition" want.

There is a second, genuinely different task that shares the same words, and the two are worth keeping straight. Speech recognition (ASR) answers what was said. Speaker recognition, also called voice biometrics, answers who said it — matching a voice against known speakers to verify an identity, the way a bank might confirm a caller by their voiceprint. A system can do one, the other, or both, but they are distinct problems: transcribing a meeting does not require knowing who anyone is, and confirming your identity does not require transcribing what you said. The rest of this page is about ASR unless it says otherwise.

How It Works

Speech is a continuous pressure wave, and text is a string of discrete symbols. ASR is the job of getting from the first to the second, and every system does it in roughly the same three moves: capture the sound, boil it down to features, and decode those features into words.

Capture. A microphone samples the sound wave thousands of times a second. Speech recognition almost always uses a 16 kHz sampling rate, and that number is not arbitrary. The Nyquist-Shannon theorem says a sampled signal can only faithfully represent frequencies up to half its sampling rate, so 16 kHz captures everything below 8 kHz — which covers essentially all of the frequency content that makes speech intelligible. Sampling faster mostly adds data without adding useful signal for this task.

Features. The raw waveform is redundant and noisy, so it is compressed into a compact time-frequency representation — historically mel-frequency cepstral coefficients (MFCCs), and in modern systems a mel-spectrogram or features learned by the network itself. The effect is the same: a stream of vectors describing how energy is distributed across pitch, several times per word, that a model can consume.

Decode. The model maps that feature stream to text. This is where the field changed shape. For decades the standard was a pipeline of separate hand-built parts: an acoustic model (long built on hidden Markov models with Gaussian mixtures, the "HMM-GMM" era) that scored which speech sounds — phonemes, of which English has around 40 — were present, a pronunciation dictionary mapping those sounds to words, and a language model scoring which word sequences are plausible. A search algorithm then stitched the pieces into the most likely sentence.

The modern approach is end-to-end: one deep neural network, usually a Transformer, trained to go straight from audio to text with no separate pronunciation dictionary. OpenAI's Whisper (Radford et al., 2022) is the well-known example — a transformer encoder-decoder trained on 680,000 hours of audio collected from the web, with the largest model at 1550M (1.55 billion) parameters. Learning acoustics, pronunciation and language jointly from that much data is what lets one model transcribe many languages and shrug off accents and noise that broke the old hand-tuned pipelines.

Measuring accuracy: Word Error Rate

You cannot say a transcript is "95% right" just by eyeballing it, because there are three distinct ways to be wrong: swap a word (substitution), drop one (deletion) or hallucinate an extra one (insertion). The standard metric, Word Error Rate (WER), counts all three against the length of the correct reference transcript:

WER = (S + I + D) / N

where S, I and D are the number of substitutions, insertions and deletions needed to turn the system's output into the reference, and N is the number of words in the reference. Lower is better; 0% is a perfect transcript, and WER can exceed 100% when insertions pile up.

A worked example. Take a 10-word reference and a hypothesis with one of each error type:

reference:  the meeting is scheduled for  three thirty     on friday afternoon
hypothesis: the meeting    scheduled four three thirty PM   on friday afternoon
                        (del "is")    (sub for→four)  (ins "PM")

That is one substitution, one deletion and one insertion against ten reference words, so WER = (1 + 1 + 1) / 10 = 30%. The number to anchor against is human performance: professional transcribers make roughly 5% word errors on conversational telephone speech (IBM measured 5.1% on the Switchboard benchmark; Saon et al., 2017), and that ~5% is the bar ASR is judged against.

Types

The honest top-level split is the one from the Definition — the two tasks people call "voice recognition":

• Speech recognition (ASR / speech-to-text) answers what was said and produces a transcript. This is the dominant meaning.
• Speaker recognition (voice biometrics) answers who said it, matching a voice against enrolled speakers to verify or identify a person.

Within ASR, the other real distinction is how much a system knows about the voice in advance. Speaker-dependent systems are tuned to one person's voice and were common when compute was scarce and enrollment (reading a training script aloud) bought a lot of accuracy. Speaker-independent systems work for anyone out of the box, which is what every modern assistant and dictation service has to be — and reaching that generality without losing accuracy is exactly what large-scale training on diverse voices buys.

Real-World Applications

The clearest place ASR shows up is dictation and transcription: talk-to-type on phones, automatic captions on YouTube and video calls, and meeting-notes tools that produce a searchable transcript. Open models like Whisper made this cheap enough to embed almost anywhere, which is why transcription features appeared across so many apps at once.

Voice assistants — Siri, Alexa, Google Assistant — use ASR as their front door: the speech-to-text step converts your words before any natural language processing works out what you meant and a text-to-speech system speaks the reply. Accessibility is a long-standing use, from live captioning for deaf and hard-of-hearing users to voice control for people who cannot use a keyboard. In healthcare, ambient documentation tools listen to a clinician-patient conversation and draft the visit notes, saving time spent typing into records.

The speaker-recognition sense has its own deployments, mostly in authentication: call centers and banks that confirm a caller by their voiceprint rather than a password. It is the same audio input as ASR but a completely different question — identity, not content.

Challenges

A single WER number is seductive and misleading, because the conditions it was measured under matter enormously. A system reported at a few percent WER on clean, read-aloud speech can be several times worse in the situations people actually use it:

• Noise and distance. Background chatter, traffic, or a phone held across the room degrade the signal the model never cleanly hears. Far-field audio (a smart speaker across a kitchen) is much harder than a microphone at your mouth.
• Accents and dialects. A model trained mostly on one accent under-performs on others, which makes WER a fairness issue, not just an accuracy one — the technology works less well for some groups of speakers than others.
• Code-switching. Speakers who mix languages mid-sentence break systems that assume one language at a time.
• Domain jargon and names. Medical terms, product names and unusual proper nouns are rare in training data, so the model falls back to a common-sounding word it has seen more often.

Two problems are intrinsic to the audio rather than the conditions. Homophones — "their" versus "there," "to" versus "two" — are acoustically identical, so only surrounding context can disambiguate them, and a weak language model will guess wrong. And raw speech carries no punctuation or capitalization: sentence boundaries, commas and proper-noun capitals all have to be inferred, which is why unpunctuated ASR output can be correct word-for-word yet hard to read.

Code Example

Here is WER computed the standard way — as a word-level edit distance between reference and hypothesis — recovering the substitution, insertion and deletion counts by walking the dynamic-programming table back. Running it on the worked example above reproduces the 30% by machine rather than by hand:

def word_error_rate(reference, hypothesis):
    ref, hyp = reference.split(), hypothesis.split()
    n, m = len(ref), len(hyp)
    # edit-distance table; each substitution, insertion or deletion costs 1
    d = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1):
        d[i][0] = i
    for j in range(m + 1):
        d[0][j] = j
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if ref[i - 1] == hyp[j - 1]:
                d[i][j] = d[i - 1][j - 1]
            else:
                d[i][j] = 1 + min(d[i - 1][j - 1],  # substitution
                                  d[i][j - 1],      # insertion
                                  d[i - 1][j])      # deletion
    # walk the table back to count each error type
    i, j, S, I, D = n, m, 0, 0, 0
    while i > 0 or j > 0:
        if i > 0 and j > 0 and ref[i - 1] == hyp[j - 1]:
            i, j = i - 1, j - 1
        elif i > 0 and j > 0 and d[i][j] == d[i - 1][j - 1] + 1:
            S += 1; i, j = i - 1, j - 1
        elif j > 0 and d[i][j] == d[i][j - 1] + 1:
            I += 1; j -= 1
        else:
            D += 1; i -= 1
    return (S + I + D) / n, S, I, D, n

reference = "the meeting is scheduled for three thirty on friday afternoon"
hypothesis = "the meeting scheduled four three thirty PM on friday afternoon"

wer, S, I, D, N = word_error_rate(reference, hypothesis)
print(f"S={S}  I={I}  D={D}  N={N}")
print(f"WER = ({S} + {I} + {D}) / {N} = {wer:.0%}")

Output:

S=1  I=1  D=1  N=10
WER = (1 + 1 + 1) / 10 = 30%

Production systems use a library such as jiwer and normalize text first — lower-casing, expanding numbers, stripping punctuation — because otherwise "30%" versus "thirty percent" or "Friday" versus "friday" would count as errors the listener would never notice.