Definition
Audio processing is how a computer turns sound — a continuous pressure wave moving through air — into numbers, and then into a representation a model can actually learn from. The pipeline is short and almost universal: a microphone measures air pressure thousands of times per second, those measurements are cut into overlapping slices of about 25 milliseconds, each slice is converted from a wiggle over time into a profile of frequencies, and the result is a 2-D image of sound that a neural network handles much the way it handles a photograph.
The reason that pipeline exists is a size problem. One second of speech recorded at 16 kHz is 16,000 numbers. The sentence it encodes might be a dozen words — a few dozen tokens of text. The same linguistic content arrives in three orders of magnitude more numbers, spread thin, with no obvious boundaries between the units that matter. Nearly every idea in this field is a way of closing that gap without throwing away the information a listener would have used.
How It Works
Sampling turns a wave into a list
Sound in the air is continuous: at every instant there is a pressure, and between any two instants there are infinitely many more. A computer cannot store that, so an analogue-to-digital converter measures the pressure at a fixed rate and keeps the measurements. CD audio uses 44,100 measurements per second per channel — 44.1 kHz — with each measurement stored as a 16-bit integer.
That fixes the size of everything downstream. One minute of CD-quality stereo is 44,100 x 60 x 2 channels x 2 bytes = 10,584,000 bytes, about 10.6 MB. One minute of 16 kHz mono speech is 16,000 x 60 x 2 = 1.92 MB, roughly a fifth as much, and an hour of it is about 115 MB before any compression. Bit depth sets the other axis: 16 bits gives 65,536 possible levels per sample and about 96 dB of dynamic range, which is why quiet passages in 16-bit recordings sit comfortably above the noise floor and 8-bit recordings hiss.
Nyquist decides what you can hear before you hear it
The single most load-bearing idea in audio is the Nyquist–Shannon sampling theorem, and most introductions skip it. It says that a sample rate of f captures frequencies only up to f/2, and that below that ceiling the original continuous wave can be reconstructed exactly. The sample rate is therefore not a quality dial. It is a decision about which sounds exist at all.
Everything above the ceiling does not simply vanish — it folds back and appears as a frequency that was never there. Sample a 9 kHz tone at 16 kHz and it comes out as a 7 kHz tone, because 16,000 − 9,000 = 7,000. This is aliasing, it is not recoverable after the fact, and it is why every converter runs a low-pass anti-alias filter before it samples anything.
Read the standard rates as Nyquist decisions and they stop looking arbitrary:
- 44.1 kHz for music puts the ceiling at 22.05 kHz, just above the roughly 20 kHz top of human hearing, with a little room for the anti-alias filter to roll off.
- 16 kHz for speech systems puts it at 8 kHz, which is where almost all of the information that distinguishes one phoneme from another lives.
- 8 kHz for traditional telephony (ITU-T G.711) puts it at 4 kHz, and the phone band is narrower still at 300–3,400 Hz. Sibilants like /s/ and /f/ carry much of their energy above 4 kHz, so the phone throws away exactly what tells them apart — which is why spelling alphabets ("S as in Sierra, F as in Foxtrot") exist for phone calls and not for conversations in a room.
Framing and the Fourier transform: the field's central move
A raw waveform is a bad input for two reasons. It is enormously long, and the thing a listener perceives — pitch, timbre, which vowel this is — is a property of the frequencies present over a short window, not of any single sample. A phoneme lasting 80 ms spans 1,280 samples at 16 kHz; no individual one of them means anything.
So the signal is cut into short overlapping frames and each frame is converted to the frequency domain with a Fast Fourier Transform. The classic setting is a 25 ms window with a 10 ms hop: at 16 kHz that is a 400-sample window (usually zero-padded to a 512-point FFT) advancing 160 samples at a time. The overlap matters because a sound that starts mid-window would otherwise be split across two frames and blurred in both.
Work the arithmetic, because it is the whole justification. A 10 ms hop means 100 frames per second, so one second of audio becomes 100 columns instead of 16,000 samples — a 160x reduction in sequence length. For a transformer, whose self-attention cost grows with the square of sequence length, 160x shorter is roughly 25,600x less attention computation. The 512-point FFT yields 257 distinct frequency bins spaced 31.25 Hz apart, so each column starts life as 257 numbers describing what that 25 ms of sound was made of. Stack the columns and you have a spectrogram: time across, frequency up, energy as brightness. At that moment a 1-D signal problem has become a 2-D image problem, and every convolutional and transformer technique built for vision applies directly.
Mel scaling adds the ear
Those 257 linear bins are not how anyone hears. Human pitch discrimination is roughly logarithmic: the gap between 100 Hz and 200 Hz is an octave and unmistakable, while the gap between 10,000 Hz and 10,100 Hz is inaudible, even though both are 100 Hz wide. The mel scale, commonly written as m = 2595 · log₁₀(1 + f/700), warps the frequency axis to match, and a bank of triangular filters spaced evenly on that warped axis sums the linear bins into far fewer perceptual ones.
Eighty mel bins is the common choice: 257 numbers per frame become 80, weighted the way an ear weights them. OpenAI's Whisper uses exactly this front end — 16 kHz audio, 25 ms windows, 10 ms stride, an 80-channel log-mel spectrogram — on 30-second chunks, which come to 3,000 frames that a strided convolution halves to 1,500 encoder positions before any attention happens. Take the logarithm of the filter outputs and you have matched perception on the loudness axis too, since loudness is also roughly logarithmic. That log-mel spectrogram, in some variant, is the input to a large share of all audio models in production.
From designed features to learned ones
Until about 2018 the front end kept going: MFCCs applied a further cosine transform to the log-mel values to decorrelate them, and a recogniser typically saw 13 to 40 coefficients per frame plus their first and second time derivatives. Every one of those steps was designed by hand, decades before the models that consumed them.
Self-supervised learning replaced most of that. Models like wav2vec 2.0 and HuBERT are trained on unlabelled audio by masking part of the signal and predicting what was hidden — the same trick that pre-trains language models, applied to sound. wav2vec 2.0 was pre-trained on 53,000 hours of unlabelled speech, and after fine-tuning on ten minutes of labelled audio it reached word error rates in the single digits, a regime that hand-designed features and supervised training never came close to. The practical consequence is that the expensive resource shifted from labelled transcripts to raw recordings, which are abundant, and that the representation a model uses is now mostly learned rather than specified.
Types
Two independent splits organise the field, and confusing them is a common source of muddle.
The first is what the sound is, and it corresponds to three research communities with their own datasets, conferences and assumptions. Speech is highly structured, carries language, and has decades of phonetics behind it. Music is polyphonic by design — many sources deliberately overlapping in time and tuned to each other — and its questions are about key, tempo, structure and similarity, a field usually called music information retrieval. Environmental sound is everything else: a door, a cough, a bearing about to fail, birdsong. It has the least structure and the fewest labels, which is why it leans hardest on unsupervised methods.
The second is what you do to it. Analysis takes audio in and emits a symbol or a label — a transcript, a genre, a speaker identity, a fault alarm. Enhancement takes audio in and emits better audio: denoising, echo cancellation, dereverberation, source separation. Generation emits audio from something that was not audio, which is the territory of text-to-speech and music generation and is covered there rather than here. The three make different demands: analysis can afford to discard information, enhancement must preserve everything it is not deliberately removing, and generation has no reference signal to compare against at all.
Real-World Applications
Noise suppression and echo cancellation, the invisible deployment. Every video call you have taken runs both. Acoustic echo cancellation solves a specific problem: your speaker's output reaches your own microphone and returns to the far end as an echo of their own voice, delayed by the round trip. Because the device already knows exactly what it played, it can predict how that signal was filtered by the room and subtract it — a problem that gets hard when someone moves a laptop and the room response changes mid-call. Neural noise suppression sits alongside it, separating voice from keyboard clatter and traffic. Both must run within the conversational latency budget, which ITU-T G.114 puts at about 150 ms one-way.
Wake words on a power budget. "Hey Siri", "Alexa" and "OK Google" are keyword spotting, and the engineering constraint is that a phone cannot run a large model continuously without destroying battery life. Apple has described a two-stage design for this: a small detector — five layers of 32 hidden units — runs permanently on a low-power always-on processor, and only when it fires does the main processor wake a larger network to confirm. The architecture is also the privacy story, because nothing is transmitted until the second stage agrees.
Audio fingerprinting for content identification. Shazam's published algorithm finds peaks in the spectrogram, hashes pairs of nearby peaks into compact codes, and matches those codes against a reference database. Peaks are chosen precisely because they survive noise, compression and a bad phone microphone, which is why the trick works in a loud bar. The same principle scales to copyright matching, where uploads are checked against a reference catalogue of recordings.
Source separation. Systems like Demucs and Spleeter split a finished mix back into vocals, drums, bass and other stems. Producers use this for remixing and karaoke; the same models underpin research on assisted listening, where the goal is to pull one talker out of a crowded room.
Speech recognition and voice interfaces. Everything above is the front end for voice recognition, which covers acoustic modelling, decoding and word error rates in detail. Increasingly the front end feeds a multimodal model directly, so audio, text and images share one representation instead of being converted between formats.
Industrial and medical acoustic monitoring. A failing pump, valve or bearing changes its sound before it changes anything a technician would notice, and microphones are cheap enough to put everywhere. The DCASE challenge series has run an anomalous-sound-detection task on machine recordings for exactly this. Note the structural constraint: you have thousands of hours of healthy machines and almost no recordings of failures, so the task is anomaly detection against a model of normal, not classification. Digital stethoscopes and cough analysis face the same shape of problem in medicine.
Key Concepts
- Sample rate and Nyquist ceiling: the rate you choose sets the highest frequency that exists in your data at f/2. Everything above it aliases into a false lower frequency and is unrecoverable.
- Bit depth: 65,536 levels at 16 bits, about 96 dB of dynamic range. It governs how quiet a sound can be before it vanishes into quantisation noise, not which frequencies survive.
- Window and hop: 25 ms and 10 ms. The first sets the time–frequency trade — longer resolves pitch better and onsets worse — and the second sets the frame rate, hence the sequence length a model sees.
- Mel filterbank: a logarithmic warping of the frequency axis matching human pitch perception, typically collapsing 257 FFT bins to 80.
- MFCC: the older hand-designed feature, a cosine transform of log-mel energies, usually 13–40 coefficients per frame. Still common in embedded systems, largely displaced by learned features elsewhere.
- Audio embedding: one fixed-length vector summarising a whole clip, learned by a network. What makes similarity search, clustering and retrieval over sound possible at all.
Challenges
Far-field audio and reverberation. A microphone across the room does not receive a quieter copy of what a close microphone receives; it receives the sound plus dozens of delayed reflections off walls, ceiling and furniture. The standard measure is RT60, the time for sound to decay by 60 dB — around 0.5 seconds in a typical office, longer in a hall. At 16 kHz, a 0.5-second RT60 means each sample is smeared across the next 8,000 samples, so the sharp onsets that mark consonants arrive blunted and overlapping with what came before. This is the single largest gap between benchmark accuracy and shipped-product accuracy.
Overlapping speakers. The cocktail-party problem is hard for a reason that is not about model quality: with one microphone and two talkers, each sample is one number containing the sum of two unknowns. The problem is underdetermined at every instant, and the only way out is a strong learned prior about what a single voice looks like. Multiple microphones help by adding spatial information, which is why smart speakers have arrays rather than one capsule, but conversational audio with genuine overlap remains far harder than the read speech most benchmarks use.
Accent, dialect and language coverage. Training data is distributed nothing like speakers are. Whisper's 680,000 hours break down as roughly 438,000 hours of English, 125,000 hours of translation data, and 117,000 hours spread across 96 other languages — an average of about 1,220 hours each, some 360x less than English gets. Under-represented accents within well-resourced languages show the same pattern in miniature, and the result is a system that works noticeably worse for some users than others while reporting one headline accuracy number.
Evaluation of things that are not transcripts. Word error rate gives speech recognition a clean target. Enhancement has no equivalent: there is no loss function for "sounds cleaner", and objective metrics correlate imperfectly with what listeners prefer, so the field still falls back on expensive human listening tests that are not comparable between studies.
Always-on microphones. The privacy problem is not hypothetical or solvable purely by policy, and the mitigations are architectural — on-device wake-word detection, short retention, edge inference — each of which costs accuracy or capability relative to sending everything to a server.
Future Trends
The clearest direction is the front end dissolving into the model. The mel spectrogram is a hand-designed compression that predates the networks consuming it, and models that operate on learned discrete audio tokens or on raw waveforms are steadily eroding the case for it — the same shift from designed to learned features that already happened one layer up when MFCCs gave way to self-supervised representations.
The second is audio becoming one input among several rather than a separate pipeline. When a single multimodal model takes sound, text and images into a shared representation and emits any of them, "audio processing" stops being a stage that hands its output to something else and becomes a property of how the model perceives. What survives that consolidation is everything upstream of any model: sampling, Nyquist, microphone arrays and room acoustics are physics, and no architecture change makes them go away.
Code Example
The whole front end in a dozen lines, with the arithmetic visible at each step:
import librosa
# 1. Sampling. Load at 16 kHz mono: the Nyquist ceiling is now 8 kHz.
y, sr = librosa.load("utterance.wav", sr=16000, mono=True)
print(y.shape, sr) # (48000,) 16000 -> 3.0 seconds, 48,000 numbers
# 2. Framing + FFT + mel filterbank.
win = int(0.025 * sr) # 25 ms window -> 400 samples
hop = int(0.010 * sr) # 10 ms hop -> 160 samples, so 100 frames/second
mel = librosa.feature.melspectrogram(
y=y, sr=sr, n_fft=512, win_length=win, hop_length=hop, n_mels=80
)
# 3. Log scaling, because loudness perception is logarithmic too.
log_mel = librosa.power_to_db(mel)
print(log_mel.shape) # (80, 301) -> 80 mel bins x 301 frames
Three seconds of audio entered as 48,000 samples and left as 301 frames of 80 values. The sequence a model must attend over shrank by 160x; the number of stored values fell by less, because each frame is now richer than a single sample. That trade — far fewer time steps, each carrying perceptually organised frequency content — is what makes the rest of audio machine learning tractable.