Definition
An autoencoder is a neural network trained to output a copy of its own input — a task that would be completely pointless if the network were not deliberately crippled in the middle. Between input and output sits a bottleneck: a layer with far fewer units than the input has values, so the data physically cannot pass through unchanged. The network is forced to throw information away, and the only way to reconstruct well after throwing information away is to have learned which parts of the data were predictable from the other parts.
So the reconstruction is not the product. The reconstruction is the exam. The compressed code sitting in the bottleneck — the latent representation, or latent vector — is the thing you actually keep and use downstream. An autoencoder given a hidden layer as wide as its input will happily reconstruct perfectly and will have learned nothing except the identity function, which is precisely why the constraint has to be imposed on purpose. Everything interesting about the architecture is a different answer to one question: how do you make copying impossible?
The two halves have names. The encoder maps the input down to the code; the decoder maps the code back up to something input-shaped. Training needs no labels — the input is its own target — which is why autoencoders sit in unsupervised learning, and why they are often described today as an early form of self-supervised learning.
How It Works
Encoder and decoder are trained together as a single network. The input x goes through the encoder to produce a code z with far fewer dimensions, the decoder turns z back into a reconstruction x̂, and a loss function measures the gap — usually mean squared error, or binary cross-entropy when the inputs are pixel intensities scaled to the range 0 to 1. Backpropagation sends that error back through the decoder and the encoder, so the encoder is being told, indirectly, which information it should have kept.
Make it concrete with MNIST. A handwritten-digit image is 28 × 28 = 784 pixels. Squeeze it to a 32-unit code and you have asked the network to describe each image with 24.5× fewer numbers (784 ÷ 32 = 24.5). What has to be discarded is everything that varies from image to image without being about the digit: the exact grey level of each individual pixel, the precise sub-pixel position of every stroke edge, scanner noise. What survives is the small set of degrees of freedom that genuinely differ between handwritten digits — which strokes are present, slant, thickness, whether a loop is closed. The digit survives the squeeze because the set of images that look like handwriting does not fill 784-dimensional space at all; it is a thin, curved sheet inside it, and 32 numbers is roughly enough to say where on that sheet you are.
One honest correction to the "24.5×" figure: MNIST pixels are 8-bit, so an image is 784 × 8 = 6,272 bits, while 32 single-precision floats are 32 × 32 = 1,024 bits. The real storage saving is 6.1×, not 24.5×. Dimension count and bit count are different quantities, and autoencoders are almost always judged on the first — which is fine when the goal is a representation, and misleading when someone quotes it as a compression ratio.
Why the non-linearity is the whole point
There is a sharp result that explains what an autoencoder is actually buying you. If the encoder and decoder are each a single linear layer with no activation function, and the loss is squared error, then the optimal solution spans exactly the same subspace as the top-k principal components (Bourlard and Kamp, 1988; Baldi and Hornik, 1989). It does not recover the individual PCA axes — any rotation within that subspace is equally optimal — but the subspace is identical. A linear autoencoder therefore buys nothing over PCA except a slower and less stable way to compute it.
Everything the architecture adds comes from the activation functions. PCA can only fit a flat k-dimensional plane through the data. A non-linear encoder can fit a curved k-dimensional surface, which is the shape most real data actually has. Hinton and Salakhutdinov's 2006 Science paper made this visible: a deep autoencoder with layers of 784-1000-500-250-30 produced markedly sharper MNIST reconstructions from a 30-number code than 30-component PCA did from the same data. When you see an autoencoder recommended over PCA for dimensionality reduction, curvature is the entire argument.
Types
The variants are not decorative. Each one is a different mechanism for stopping the network from cheating, and they solve genuinely different problems.
Undercomplete autoencoder
The baseline: the code has fewer units than the input, and that narrowness is the only constraint. Simple, and it works — but the safety margin is thin. A deep enough encoder and decoder can learn a nearly-lossless code for the training set specifically, effectively memorising it, which is overfitting with an unusually well-disguised symptom (the reconstructions look great).
Denoising autoencoder
Rather than narrowing the pipe, change the task. Corrupt the input — mask 25% to 50% of the values to zero, or add Gaussian noise — and ask the network to output the clean original (Vincent et al., ICML 2008). Now copying is impossible in principle rather than merely difficult: the information about the masked values is not present in the input at all, so it has to be inferred from a learned model of how the values relate to one another. This is why denoising autoencoders learn better features than plain ones even when the code is not narrower than the input. It is also a direct ancestor of masked language modelling — BERT-style masked token prediction is a denoising autoencoder objective applied to text.
Sparse autoencoder
Constrain by activity instead of by width. Add an L1 penalty on the code's activations so that only a small fraction of units may fire for any one input. The code can then be far wider than the input and still learn something, because the cost is paid per active unit rather than per unit. This variant has had an unexpected second life in interpretability: sparse autoencoders are now the main tool for decomposing a language model's internal activations into human-readable features, with Anthropic's 2024 "Scaling Monosemanticity" work training sparse autoencoders on Claude 3 Sonnet at 1 million, 4 million and 34 million features.
Variational autoencoder (VAE)
The generative variant, and the reason for the extra machinery is worth stating precisely. In an ordinary autoencoder the latent space has no imposed structure: the encoder may scatter training points anywhere it likes, so a point sampled between two known codes has no reason to decode to anything sensible, and sampling a code at random almost certainly decodes to noise. A VAE (Kingma and Welling, 2013) makes the encoder emit a distribution — a mean and a variance — rather than a point, samples the code from it, and adds a KL-divergence term pulling those distributions toward a standard normal. Two things follow: nearby codes decode to similar outputs, and you can generate a new sample by drawing a code from the standard normal and running only the decoder. The cost is that the KL term competes with reconstruction quality, which is why plain VAE samples are famously blurry.
Masked autoencoder (MAE)
The modern computer-vision descendant. He et al. (2021) split an image into patches, mask 75% of them, feed only the visible 25% to the encoder, and use a lightweight decoder to reconstruct the missing pixels. Since a 224 × 224 image at 16 × 16 patches is 196 patches, the encoder processes about 49 of them — which is why pretraining runs roughly 3× faster than approaches that encode every patch. A ViT-Huge pretrained this way and then fine-tuned reached 87.8% top-1 accuracy on ImageNet-1K using ImageNet-1K data only. MAE is the clearest demonstration that the denoising idea scales.
Real-World Applications
Latent diffusion image models. The most widely deployed autoencoder in production is the one almost nobody looks at: the VAE inside Stable Diffusion and its descendants. A 512 × 512 RGB image is 512 × 512 × 3 = 786,432 numbers; the encoder compresses it to a 64 × 64 × 4 latent tensor of 16,384 numbers, a 48× reduction. The diffusion process then runs entirely in that latent space and the decoder converts the result back to pixels at the very end. This is what made high-resolution image generation affordable on consumer hardware — the expensive denoising network never touches a full-resolution image.
Neural audio codecs. SoundStream (Google, 2021) and EnCodec (Meta, 2022) are autoencoders with a quantised bottleneck, compressing 24 kHz audio to bitrates in the range of roughly 1.5 to 24 kbit/s at quality competitive with hand-designed codecs. Their discrete codes double as tokens, which is how audio gets fed into transformer-based speech and music models at all.
Industrial visual inspection. Train an autoencoder only on defect-free parts, then flag regions whose reconstruction error is high. The MVTec AD benchmark (Bergmann et al., 2019) exists specifically for this setting — 5,354 high-resolution images across 15 object and texture categories, with training sets containing no defects at all. The appeal is practical: factories have thousands of good parts and almost no labelled examples of each rare defect, which rules out ordinary supervised classification. See anomaly detection for how this compares with other approaches, and read the caveat below before deploying it.
Compact features for downstream models. Codes from a trained encoder can be fed to a classifier, a nearest-neighbour search or a clustering algorithm, in the same role an embedding plays elsewhere. This is the oldest use and the one most eroded by newer methods, since a pretrained contrastive or masked-prediction model usually gives better features for the same effort.
Challenges
Reconstruction-error anomaly detection fails more often than tutorials admit. The method assumes that a model trained on normal data will reconstruct abnormal data badly. That assumption is not reliable. A high-capacity autoencoder learns generic low-level structure — edges, smoothness, local colour statistics — that transfers perfectly well to inputs it has never seen, so anomalies come back cleanly reconstructed and the error signal disappears. The same failure shows up in likelihood terms: Nalisnick et al. (2019) found that deep generative models, VAEs included, trained on CIFAR-10 assign higher likelihood to SVHN images than to CIFAR-10 itself. If you use this technique, keep the bottleneck deliberately narrow, calibrate the threshold on a held-out set that contains real defects rather than on the training loss, and compare against a simpler baseline before believing the result.
Squared error produces blurry outputs. When several reconstructions are plausible, minimising mean squared error rewards predicting their average, and the average of several sharp images is a soft one. This is why VAE samples look smeared, and why production systems that need sharp output pair the autoencoder with something else — an adversarial or perceptual loss term, or a diffusion process running in the latent space.
Nothing makes the code interpretable. The objective asks for reconstructability and nothing more. There is no pressure for individual latent units to correspond to anything a human would name, and in a plain autoencoder they generally do not. Sparse and disentanglement-oriented variants add that pressure explicitly, at a cost in reconstruction quality.
The bottleneck width has no principled setting. Too narrow and you destroy signal you needed; too wide and you approach the identity function and learn nothing. It also interacts with depth and regularization, so it cannot be tuned in isolation — which is exactly what the code example below makes visible.
Where they are, and are not, the right tool. As a general-purpose representation learning method, plain autoencoders were largely superseded during the late 2010s and early 2020s by contrastive objectives and masked-prediction pretraining, which produce features that transfer better. They are still the first choice in four narrower places: as the compression layer inside a larger generative system such as latent diffusion or a neural codec; as sparse autoencoders for interpretability; for anomaly detection when only normal examples exist; and for non-linear dimensionality reduction where PCA's flat subspace is the wrong shape for the data. Describing them as the state of the art in representation learning would be about a decade out of date.
Code Example
This trains an undercomplete autoencoder in NumPy on data that genuinely lives on an 8-dimensional curved surface inside 64-dimensional space, at three bottleneck widths. The point is the trade-off, not the architecture — run it and watch the error at width 64, where there is no bottleneck at all.
import numpy as np
rng = np.random.default_rng(0)
# 4,096 points on an 8-dimensional CURVED surface embedded in 64-D space
X = np.tanh(rng.normal(size=(4096, 8)) @ rng.normal(size=(8, 64)))
def train(k, steps=6000, lr=0.5):
W1 = rng.normal(size=(64, k)) * 0.1 # encoder: 64 inputs -> k-unit code
W2 = rng.normal(size=(k, 64)) * 0.1 # decoder: k-unit code -> 64 outputs
for _ in range(steps):
code = np.tanh(X @ W1) # the bottleneck: k numbers per point
recon = code @ W2
g = 2 * (recon - X) / X.size # d(MSE) / d(recon)
gW1 = X.T @ ((g @ W2.T) * (1 - code ** 2))
W2 -= lr * (code.T @ g)
W1 -= lr * gW1
return ((np.tanh(X @ W1) @ W2 - X) ** 2).mean()
for k in (2, 8, 64):
print(f"bottleneck {k:2d} units -> reconstruction MSE {train(k):.4f}")
Output:
bottleneck 2 units -> reconstruction MSE 0.5172
bottleneck 8 units -> reconstruction MSE 0.1099
bottleneck 64 units -> reconstruction MSE 0.0027
The data's own variance is 0.7212, which is the error you would get by ignoring the input entirely and predicting the mean. So a 2-unit code explains 28% of the variance, an 8-unit code 85%, and a 64-unit code 99.6% — and that last figure is the warning, not the achievement. At width 64 the "bottleneck" is as wide as the input, nothing had to be discarded, and the network has essentially learned to pass the data through. Its reconstruction is the best of the three and its representation is the most useless of the three. The 8-unit run, which matches the true dimensionality of the surface the data lies on, is the one that learned something: it reproduces most of the data from one-eighth of the numbers because it worked out what shape the data has.