Dimensionality Reduction (DR)

Definition

Dimensionality reduction rewrites data that is described by many numbers per item — 784 pixel values for a handwritten digit, 20,000 gene counts for a single cell, 3,072 floats for a text embedding — using far fewer numbers, chosen so the structure you care about survives the cut. You discard dimensions on purpose because high-dimensional space is hostile to the algorithms that find structure: among 1,000 points scattered uniformly in 100 dimensions, the farthest point from a query is only 1.43× as far away as the nearest one, so "nearest neighbour" stops being a category that distinguishes anything. The arithmetic is worked below.

Which method to reach for has a short answer. PCA is the default whenever the output feeds another algorithm: it is fast, deterministic, invertible, applies to new data as a single matrix multiply, and it tells you exactly what fraction of the data you kept. t-SNE and UMAP are for one job — drawing a two-dimensional picture for a human to look at. They are not a preprocessing step, their axes have no units, and, as the rest of this page argues, most of what people read off their plots is not there. When the data sits on a curved surface that no flat projection can follow and you still need a usable representation rather than a picture, that is what an autoencoder is for.

One distinction worth keeping straight: this is feature extraction, which invents new composite coordinates out of all the originals, and it is not feature selection, which keeps a subset of the original columns untouched. Extraction usually preserves more information per dimension. Selection keeps columns that still mean what they meant, which is what you want when a human has to act on the answer.

How It Works

Two facts sit underneath every method here. The first is the curse of dimensionality: as the number of dimensions grows, the space empties out and distance stops discriminating. The second is the manifold assumption: real data does not fill the space it is stored in — photographs of faces occupy a vanishingly thin sliver of all possible pixel arrays — so a faithful description in far fewer numbers usually exists. The first says you must reduce. The second says you can.

The curse of dimensionality, in arithmetic

Take a cube of side 1 in d dimensions and ask what fraction of its volume lies within 0.01 of the surface. The interior that escapes the shell is a smaller cube of side 0.98, with volume 0.98^d, so the shell's share is 1 − 0.98^d:

At 100 dimensions, 87% of the cube is within a hair of an edge. There is no interior left. Every point is an outlier in some coordinate, and "the middle of the data" has ceased to be a place.

The consequence that actually breaks software follows from that emptiness. Scatter 1,000 points uniformly in the unit cube, pick a query point, and measure the distance from the query to its nearest and farthest neighbour among those 1,000. The code example at the end runs this; the medians over 100 queries are:

In two dimensions the nearest neighbour is roughly seventy times closer than the farthest, and picking it out is trivially meaningful. In a thousand dimensions everything sits at a distance of about 12 or 13 from everything else, and the worst match in the whole set is only 12% further away than the best one. This is what people mean by distance concentration, and it is why the honest version of the curse is not "computation gets slow" but "your similarity measure stops ranking anything." It takes out k-nearest-neighbour classification, the assignment step in k-means, any DBSCAN-style radius threshold, and the recall of an approximate vector search index, all through the same mechanism.

The reason this is survivable is the manifold assumption. Real data is not uniform — that is the entire content of the assumption — and what governs the damage is the intrinsic dimension of the surface the data lies on, not the number of columns you happen to store. A 4,096-pixel photograph of a face varies along perhaps a few dozen real degrees of freedom: pose, expression, lighting, identity. Dimensionality reduction is the attempt to find coordinates for that surface, and it works exactly to the extent that the surface is genuinely low-dimensional.

What PCA actually does

Principal Component Analysis asks a geometric question with a closed-form answer. Consider every possible direction you could draw through the centre of the data, project all the points onto it, and measure how spread out the projections are. The direction with the greatest spread is the first principal component. Now restrict yourself to directions perpendicular to that one and ask again: that is the second component. Repeat until you run out of dimensions. The components are orthogonal by construction, so they carve the total variance into non-overlapping shares, and the share belonging to each is its explained variance ratio. In practice you compute all of this at once as the singular value decomposition of the centred data matrix.

Because the shares are non-overlapping and sum to 1, you can add them up and read k off the resulting cumulative curve rather than guessing. Two real datasets show how differently that curve can behave. On Fisher's iris data — 150 flowers, four measurements each — the first component carries 92.46% of the variance and the second 5.30%, so a two-dimensional scatter plot of iris hides just 2.2% of the variance and is an essentially honest picture. On MNIST's 60,000 training images of handwritten digits, each 28 × 28 = 784 pixels, the first two components carry only 9.70% and 7.10%, a cumulative 16.8% — which is precisely why a 2-D PCA plot of MNIST is a smear, and why people reach for t-SNE on this kind of data. Keep going up the curve, though, and 90% of the variance arrives at 87 components, 95% at 154, and 99% at 331. Compressing 784 pixels to 154 numbers, a 5.1× reduction, costs 5% of the variance.

The contrast is the lesson: the right number of components is a property of the dataset, not a setting you choose by taste. Two components suffice for iris and are useless for MNIST.

Two properties make PCA the safe default. It is a linear map — a single 784 × 154 matrix, in the MNIST case — so applying it to data it has never seen costs one multiplication, which means it can be fitted on a training set and applied to a test set like any other transform. And it is invertible: multiply the reduced representation back through the components and you get an approximate reconstruction whose squared error equals, exactly, the variance you discarded. That is an unusually strong guarantee, and it is the property that the methods in the next section give up.

One prerequisite: PCA maximises variance, and variance carries units. Record a column in millimetres instead of metres and its variance grows by a factor of 10⁶, which will drag the first component onto it regardless of whether that column matters. Unless every feature is already in the same unit — pixel intensities, for instance — standardise each to zero mean and unit variance first. On iris the choice is visible: on the raw centimetre measurements the first two components carry 97.8% of the variance, on the standardised version 95.8%. The same trap applies to clustering, for the same reason.

Types

Linear versus non-linear is not a taxonomic nicety here. It determines what you are permitted to do with the output, and confusing the two is the most expensive mistake on this page.

Linear: PCA, SVD, and truncated SVD

Every new coordinate is a fixed weighted sum of the original ones, which is what buys the guarantees above: one matrix, applicable to new points, invertible, with the loss measurable in advance. It also means you can inspect the weights — the loadings — and say which original features drive a given component, so the axes retain a thread back to something nameable.

Truncated SVD is the same decomposition applied without centring the data, which matters for sparse matrices where subtracting the mean would fill in every zero and destroy the sparsity. This is what latent semantic analysis has always been: Deerwester et al. (1990) factorised a term–document matrix down to a few hundred dimensions, and the resulting space put car and automobile near each other despite their sharing no characters — the direct ancestor of modern semantic search.

What linear methods cannot do is follow curvature. On the standard "Swiss roll" — a sheet rolled into a spiral — the shortest path along the sheet between two points may be long while their straight-line distance is short, because the roll brings distant parts of the sheet close together. Any flat projection superimposes them. The fix is either a non-linear method or an autoencoder, whose non-linear encoder can fit a curved surface where PCA can only fit a flat one.

Non-linear and manifold: t-SNE and UMAP

t-SNE (van der Maaten and Hinton, 2008) converts pairwise distances in the original space into probabilities that one point would pick another as its neighbour, then shuffles points around a 2-D canvas until the same probabilities hold there, using a heavy-tailed Student-t distribution in the low-dimensional space so that distant points are not crushed together. UMAP (McInnes et al., 2018) builds a weighted k-nearest-neighbour graph and optimises a low-dimensional layout that preserves it. Different mathematics, same contract: both preserve who your neighbours are, and neither preserves how far apart anything is.

That contract is narrower than almost every reader of such a plot assumes, and here is what it rules out:

• The axes mean nothing. They have no units, no ordering and no interpretation. The plot can be rotated or flipped without changing a single claim it makes.
• Cluster size means nothing. t-SNE rescales distance according to local density, so a tightly packed group and a diffuse one are drawn at comparable size. A blob being large is not evidence that its members are varied.
• The gap between two clusters means nothing. Two blobs drawn far apart are not more different than two drawn close together; the algorithm was never asked to make that true, and it does not optimise it.
• Apparent clusters can be pure artefact. Wattenberg, Viégas and Johnson showed in Distill (2016) that t-SNE at low perplexity carves random Gaussian noise into clean, convincing, entirely meaningless clumps. Any structural claim from a single run at a single parameter setting is unsupported.
• There is no honest inverse and, for t-SNE, no transform. t-SNE has no mapping you can apply to a new point; adding one means re-running the optimisation, and the whole picture may rearrange. UMAP offers an approximate transform, which is better but is not PCA's exact matrix.

Chari and Pachter (2023) pushed this further in single-cell biology, showing that 2-D embeddings distort both local and global structure enough that conclusions drawn from their geometry are unreliable. None of this makes t-SNE or UMAP bad. It makes them plotting tools with a single legitimate reading: these points were neighbours in the original space. The practical rule that falls out is that you cluster in the reduced-but-still-quantitative space — say 50 principal components — and use the 2-D layout only to colour and display the result. Clustering the 2-D coordinates themselves recovers the layout algorithm's artefacts and reports them as biology.

Real-World Applications

Single-cell RNA sequencing. A single-cell experiment measures on the order of 20,000 genes across tens of thousands of cells, and the standard Seurat and Scanpy pipelines reduce it twice for two different reasons. First, select roughly 2,000 highly variable genes and run PCA down to about 50 components — this is a denoising step, since most of the discarded variance is measurement noise from genes detected in a handful of cells. Then build a neighbour graph on those 50 components, cluster it, and separately run UMAP on the same 50 components to produce the figure in the paper. The division of labour is the point: the quantitative work happens in 50 dimensions and the two-dimensional plot is a display layer.

Compressing embeddings before a vector index. Embedding vectors are stored, not just computed, and the arithmetic is unforgiving: one million vectors of 3,072 float32 dimensions occupy 1,000,000 × 3,072 × 4 = 12.3 GB. Cut the same vectors to 256 dimensions and they occupy 1.02 GB — 12× less — which is often the difference between an index that fits in RAM and one that has to be served from disk, and so between millisecond and hundred-millisecond retrieval. Matryoshka Representation Learning (Kusupati et al., 2022) trains embeddings so that each prefix is itself a usable vector, making the reduction a slice rather than a projection; product quantization, the compression inside most vector search libraries, goes further by splitting the vector into sub-blocks and replacing each with a codebook index.

Correcting for ancestry in genetic association studies. A genome-wide association study compares hundreds of thousands of genetic variants between cases and controls, and if the two groups differ even slightly in ancestry, thousands of variants will correlate with the disease for reasons that have nothing to do with it. EIGENSTRAT (Price et al., 2006) runs PCA on the genotype matrix and includes the top components — commonly ten — as covariates in every association test, so anything explained by ancestry is absorbed before the disease signal is measured. That this works is not obvious, but Novembre et al. (2008) made it vivid: the first two principal components of genotype data from about 1,400 Europeans reproduce the map of Europe, with the components aligning to latitude and longitude. Here dimensionality reduction is not compression at all — it is a measurement of a confounder.

Challenges

Variance is not the same as usefulness. PCA ranks directions by spread, and spread has no relationship to the label you are trying to predict. A signal that separates two classes cleanly but with small amplitude can land in the components you discarded, and you will have destroyed the only thing you needed while preserving 95% of the variance. Linear discriminant analysis optimises class separation instead and is the supervised alternative. More generally, an explained-variance threshold optimises reconstruction; if a supervised model consumes the output, choose k by cross-validated task performance instead.

Fit the reduction inside the split, not before it. Running PCA on the full dataset and then splitting into train and test lets the test set influence the axes, and the resulting score is optimistic for reasons no amount of careful modelling downstream will fix. The reduction is part of the model and belongs inside the cross-validation loop, refitted on each training fold. This is one of the most common quiet sources of overfitting in tabular pipelines.

t-SNE and UMAP are not reproducible the way PCA is. Both are stochastic optimisations, and both have parameters that change the picture qualitatively — perplexity for t-SNE, n_neighbors and min_dist for UMAP. Two seeds give two different-looking pictures of identical data, while PCA returns the same subspace every time, up to a sign flip per axis. The working test is simple: change the seed and the parameters, rerun, and see whether the claim survives. If it does not, it was a claim about the layout.

Reduced components resist interpretation. "The third principal component increased" is not a sentence anyone can act on. PCA at least offers loadings — the weights of the original features in each component, which sometimes name themselves, as when the leading component of genotype data turns out to be geography. Non-linear methods offer nothing comparable. When the deliverable is an explanation rather than a prediction, feature selection may be the better tool even though it preserves less information, because its output is a list of things that already have names.

Code Example

This is the distance-concentration measurement from above, in about ten lines of NumPy. It is worth running rather than reading: the collapse between 2 and 100 dimensions is the single fact that motivates everything else on this page.

import numpy as np
rng = np.random.default_rng(0)

for d in (2, 10, 100, 1000):
    near, far = [], []
    for _ in range(100):                       # 100 independent query points
        X = rng.random((1000, d))              # 1,000 points, uniform in the unit cube
        q = rng.random(d)                      # the query
        dist = np.sqrt(((X - q) ** 2).sum(1))
        near.append(dist.min())
        far.append(dist.max())
    n, f = np.median(near), np.median(far)
    print(f"d={d:5d}   nearest {n:6.3f}   farthest {f:6.3f}   ratio {f / n:7.2f}")

Output:

d=    2   nearest  0.015   farthest  1.052   ratio   68.84
d=   10   nearest  0.509   farthest  1.921   ratio    3.77
d=  100   nearest  3.328   farthest  4.772   ratio    1.43
d= 1000   nearest 12.166   farthest 13.630   ratio    1.12

Note what does not change: the number of points is 1,000 in every row. The data did not get sparser because there is less of it; it got sparser because the space grew. Adding points barely helps, either — to keep the same density of 10 samples per axis you would need 10^d points, which is 10 billion at d = 10 and hopeless past that. Reducing d is the only lever that moves, which is the whole argument for doing it.