---
source: 'https://howaiworks.ai/glossary/dimensionality-reduction'
section: glossary
title: Dimensionality Reduction (DR)
description: >-
Why high-dimensional data has to be compressed, how PCA picks its axes, and
why cluster sizes and gaps in a t-SNE or UMAP plot mean nothing.
tags:
- dimensionality reduction
- unsupervised learning
- Machine Learning
- feature extraction
category: Machine Learning
datePublished: '2025-07-16'
lastUpdated: '2026-07-22'
---
# Dimensionality Reduction (DR)
> Why high-dimensional data has to be compressed, how PCA picks its axes, and why cluster sizes and gaps in a t-SNE or UMAP plot mean nothing.
## Definition
Dimensionality reduction rewrites data that is described by many numbers per item — 784 pixel
values for a handwritten digit, 20,000 gene counts for a single cell, 3,072 floats for a text
[embedding](https://howaiworks.ai/glossary/embedding) — using far fewer numbers, chosen so the structure you care about
survives the cut. You discard dimensions on purpose because high-dimensional space is hostile to
the algorithms that find structure: among 1,000 points scattered uniformly in 100 dimensions, the
*farthest* point from a query is only **1.43× as far away** as the nearest one, so "nearest
neighbour" stops being a category that distinguishes anything. The arithmetic is worked below.
Which method to reach for has a short answer. **PCA** is the default whenever the output feeds
another algorithm: it is fast, deterministic, invertible, applies to new data as a single matrix
multiply, and it tells you exactly what fraction of the data you kept. **t-SNE and UMAP** are for
one job — drawing a two-dimensional picture for a human to look at. They are not a preprocessing
step, their axes have no units, and, as the rest of this page argues, most of what people read off
their plots is not there. When the data sits on a curved surface that no flat projection can
follow and you still need a usable representation rather than a picture, that is what an
[autoencoder](https://howaiworks.ai/glossary/autoencoder) is for.
One distinction worth keeping straight: this is *feature extraction*, which invents new composite
coordinates out of all the originals, and it is not
[feature selection](https://howaiworks.ai/glossary/feature-selection), which keeps a subset of the original columns
untouched. Extraction usually preserves more information per dimension. Selection keeps columns
that still mean what they meant, which is what you want when a human has to act on the answer.
## How It Works
Two facts sit underneath every method here. The first is the curse of dimensionality: as the
number of dimensions grows, the space empties out and distance stops discriminating. The second is
the manifold assumption: real data does not fill the space it is stored in — photographs of faces
occupy a vanishingly thin sliver of all possible pixel arrays — so a faithful description in far
fewer numbers usually exists. The first says you must reduce. The second says you can.
### The curse of dimensionality, in arithmetic
Take a cube of side 1 in *d* dimensions and ask what fraction of its volume lies within 0.01 of
the surface. The interior that escapes the shell is a smaller cube of side 0.98, with volume
0.98d, so the shell's share is 1 − 0.98d:
| dimensions | volume within 0.01 of the surface |
|---|---|
| 1 | 2.0% |
| 10 | 18.3% |
| 100 | **86.7%** |
| 500 | 99.996% |
At 100 dimensions, 87% of the cube is within a hair of an edge. There is no interior left. Every
point is an outlier in some coordinate, and "the middle of the data" has ceased to be a place.
The consequence that actually breaks software follows from that emptiness. Scatter 1,000 points
uniformly in the unit cube, pick a query point, and measure the distance from the query to its
nearest and farthest neighbour among those 1,000. The [code example](#code-example) at the end
runs this; the medians over 100 queries are:
| dimensions | nearest | farthest | farthest ÷ nearest |
|---|---|---|---|
| 2 | 0.015 | 1.052 | 68.84× |
| 10 | 0.509 | 1.921 | 3.77× |
| 100 | 3.328 | 4.772 | **1.43×** |
| 1,000 | 12.166 | 13.630 | 1.12× |
In two dimensions the nearest neighbour is roughly seventy times closer than the farthest, and
picking it out is trivially meaningful. In a thousand dimensions everything sits at a distance of
about 12 or 13 from everything else, and the *worst* match in the whole set is only 12% further
away than the best one. This is what people mean by *distance concentration*, and it is why the
honest
version of the curse is not "computation gets slow" but "your similarity measure stops ranking
anything." It takes out k-nearest-neighbour classification, the assignment step in k-means, any
DBSCAN-style radius threshold, and the recall of an approximate
[vector search](https://howaiworks.ai/glossary/vector-search) index, all through the same mechanism.
The reason this is survivable is the manifold assumption. Real data is not uniform — that is the
entire content of the assumption — and what governs the damage is the *intrinsic* dimension of the
surface the data lies on, not the number of columns you happen to store. A 4,096-pixel photograph
of a face varies along perhaps a few dozen real degrees of freedom: pose, expression, lighting,
identity. Dimensionality reduction is the attempt to find coordinates for that surface, and it
works exactly to the extent that the surface is genuinely low-dimensional.
### What PCA actually does
[Principal Component Analysis](https://howaiworks.ai/glossary/principal-component-analysis) asks a geometric question with a closed-form answer. Consider every
possible direction you could draw through the centre of the data, project all the points onto it,
and measure how spread out the projections are. The direction with the greatest spread is the
**first principal component**. Now restrict yourself to directions perpendicular to that one and
ask again: that is the second component. Repeat until you run out of dimensions. The components
are orthogonal by construction, so they carve the total variance into non-overlapping shares, and
the share belonging to each is its **explained variance ratio**. In practice you compute all of
this at once as the singular value decomposition of the centred data matrix.
Because the shares are non-overlapping and sum to 1, you can add them up and read *k* off the
resulting cumulative curve rather than guessing. Two real datasets show how differently that curve
can behave. On Fisher's iris data — 150 flowers, four measurements each — the first component
carries **92.46%** of the variance and the second **5.30%**, so a two-dimensional scatter plot of
iris hides just 2.2% of the variance and is an essentially honest picture. On MNIST's 60,000
training images of handwritten digits, each 28 × 28 = 784 pixels, the first two components carry
only 9.70% and 7.10%, a cumulative **16.8%** — which is precisely why a 2-D PCA plot of MNIST is a
smear, and why people reach for t-SNE on this kind of data. Keep going up the curve, though, and
90% of the variance arrives at 87 components, 95% at **154**, and 99% at 331. Compressing 784
pixels to 154 numbers, a 5.1× reduction, costs 5% of the variance.
The contrast is the lesson: the right number of components is a property of the dataset, not a
setting you choose by taste. Two components suffice for iris and are useless for MNIST.
Two properties make PCA the safe default. It is a **linear map** — a single 784 × 154 matrix, in
the MNIST case — so applying it to data it has never seen costs one multiplication, which means it
can be fitted on a training set and applied to a test set like any other transform. And it is
**invertible**: multiply the reduced representation back through the components and you get an
approximate reconstruction whose squared error equals, exactly, the variance you discarded. That
is an unusually strong guarantee, and it is the property that the methods in the next section give
up.
One prerequisite: PCA maximises variance, and variance carries units. Record a column in
millimetres instead of metres and its variance grows by a factor of 106, which will
drag the first component onto it regardless of whether that column matters. Unless every feature
is already in the same unit — pixel intensities, for instance — standardise each to zero mean and
unit variance first. On iris the choice is visible: on the raw centimetre measurements the first
two components carry 97.8% of the variance, on the standardised version 95.8%. The same trap
applies to [clustering](https://howaiworks.ai/glossary/clustering), for the same reason.
## Types
Linear versus non-linear is not a taxonomic nicety here. It determines what you are permitted to
do with the output, and confusing the two is the most expensive mistake on this page.
### Linear: PCA, SVD, and truncated SVD
Every new coordinate is a fixed weighted sum of the original ones, which is what buys the
guarantees above: one matrix, applicable to new points, invertible, with the loss measurable in
advance. It also means you can inspect the weights — the loadings — and say which original
features drive a given component, so the axes retain a thread back to something nameable.
**Truncated SVD** is the same decomposition applied without centring the data, which matters for
sparse matrices where subtracting the mean would fill in every zero and destroy the sparsity. This
is what latent semantic analysis has always been: Deerwester et al. (1990) factorised a
term–document matrix down to a few hundred dimensions, and the resulting space put *car* and
*automobile* near each other despite their sharing no characters — the direct ancestor of modern
[semantic search](https://howaiworks.ai/glossary/semantic-search).
What linear methods cannot do is follow curvature. On the standard "Swiss roll" — a sheet rolled
into a spiral — the shortest path *along the sheet* between two points may be long while their
straight-line distance is short, because the roll brings distant parts of the sheet close
together. Any flat projection superimposes them. The fix is either a non-linear method or an
[autoencoder](https://howaiworks.ai/glossary/autoencoder), whose non-linear encoder can fit a curved surface where PCA
can only fit a flat one.
### Non-linear and manifold: t-SNE and UMAP
t-SNE (van der Maaten and Hinton, 2008) converts pairwise distances in the original space into
probabilities that one point would pick another as its neighbour, then shuffles points around a
2-D canvas until the same probabilities hold there, using a heavy-tailed Student-t distribution in
the low-dimensional space so that distant points are not crushed together. UMAP (McInnes et al.,
2018) builds a weighted k-nearest-neighbour graph and optimises a low-dimensional layout that
preserves it. Different mathematics, same contract: **both preserve who your neighbours are, and
neither preserves how far apart anything is.**
That contract is narrower than almost every reader of such a plot assumes, and here is what it
rules out:
- **The axes mean nothing.** They have no units, no ordering and no interpretation. The plot can
be rotated or flipped without changing a single claim it makes.
- **Cluster size means nothing.** t-SNE rescales distance according to local density, so a tightly
packed group and a diffuse one are drawn at comparable size. A blob being large is not evidence
that its members are varied.
- **The gap between two clusters means nothing.** Two blobs drawn far apart are not more different
than two drawn close together; the algorithm was never asked to make that true, and it does not
optimise it.
- **Apparent clusters can be pure artefact.** Wattenberg, Viégas and Johnson showed in *Distill*
(2016) that t-SNE at low perplexity carves random Gaussian noise into clean, convincing,
entirely meaningless clumps. Any structural claim from a single run at a single parameter
setting is unsupported.
- **There is no honest inverse and, for t-SNE, no transform.** t-SNE has no mapping you can apply
to a new point; adding one means re-running the optimisation, and the whole picture may
rearrange. UMAP offers an approximate transform, which is better but is not PCA's exact matrix.
Chari and Pachter (2023) pushed this further in single-cell biology, showing that 2-D embeddings
distort both local and global structure enough that conclusions drawn from their geometry are
unreliable. None of this makes t-SNE or UMAP bad. It makes them plotting tools with a single
legitimate reading: *these points were neighbours in the original space.* The practical rule that
falls out is that you cluster in the reduced-but-still-quantitative space — say 50 principal
components — and use the 2-D layout only to colour and display the result. Clustering the 2-D
coordinates themselves recovers the layout algorithm's artefacts and reports them as biology.
## Real-World Applications
**Single-cell RNA sequencing.** A single-cell experiment measures on the order of 20,000 genes
across tens of thousands of cells, and the standard Seurat and Scanpy pipelines reduce it twice
for two different reasons. First, select roughly 2,000 highly variable genes and run PCA down to
about 50 components — this is a denoising step, since most of the discarded variance is
measurement noise from genes detected in a handful of cells. Then build a neighbour graph on those
50 components, cluster it, and separately run UMAP on the same 50 components to produce the figure
in the paper. The division of labour is the point: the quantitative work happens in 50 dimensions
and the two-dimensional plot is a display layer.
**Compressing embeddings before a vector index.** Embedding vectors are stored, not just computed,
and the arithmetic is unforgiving: one million vectors of 3,072 float32 dimensions occupy
1,000,000 × 3,072 × 4 = 12.3 GB. Cut the same vectors to 256 dimensions and they occupy 1.02 GB —
12× less — which is often the difference between an index that fits in RAM and one that has to be
served from disk, and so between millisecond and hundred-millisecond retrieval. Matryoshka
Representation Learning (Kusupati et al., 2022) trains embeddings so that each prefix is itself a
usable vector, making the reduction a slice rather than a projection; product quantization, the
compression inside most
[vector search](https://howaiworks.ai/glossary/vector-search) libraries, goes further by splitting the vector into
sub-blocks and replacing each with a codebook index.
**Correcting for ancestry in genetic association studies.** A genome-wide association study
compares hundreds of thousands of genetic variants between cases and controls, and if the two
groups differ even slightly in ancestry, thousands of variants will correlate with the disease for
reasons that have nothing to do with it. EIGENSTRAT (Price et al., 2006) runs PCA on the genotype
matrix and includes the top components — commonly ten — as covariates in every association test,
so anything explained by ancestry is absorbed before the disease signal is measured. That this
works is not obvious, but Novembre et al. (2008) made it vivid: the first two principal components
of genotype data from about 1,400 Europeans reproduce the map of Europe, with the components
aligning to latitude and longitude. Here dimensionality reduction is not compression at all — it
is a measurement of a confounder.
## Challenges
**Variance is not the same as usefulness.** PCA ranks directions by spread, and spread has no
relationship to the label you are trying to predict. A signal that separates two classes cleanly
but with small amplitude can land in the components you discarded, and you will have destroyed the
only thing you needed while preserving 95% of the variance. Linear discriminant analysis optimises
class separation instead and is the supervised alternative. More generally, an explained-variance
threshold optimises reconstruction; if a supervised model consumes the output, choose *k* by
cross-validated task performance instead.
**Fit the reduction inside the split, not before it.** Running PCA on the full dataset and then
splitting into train and test lets the test set influence the axes, and the resulting score is
optimistic for reasons no amount of careful modelling downstream will fix. The reduction is part
of the model and belongs inside the [cross-validation](https://howaiworks.ai/glossary/cross-validation) loop, refitted
on each training fold. This is one of the most common quiet sources of
[overfitting](https://howaiworks.ai/glossary/overfitting) in tabular pipelines.
**t-SNE and UMAP are not reproducible the way PCA is.** Both are stochastic optimisations, and
both have parameters that change the picture qualitatively — perplexity for t-SNE, `n_neighbors`
and `min_dist` for UMAP. Two seeds give two different-looking pictures of identical data, while
PCA returns the same subspace every time, up to a sign flip per axis. The working test is simple:
change the seed and the parameters, rerun, and see whether the claim survives. If it does not, it
was a claim about the layout.
**Reduced components resist interpretation.** "The third principal component increased" is not a
sentence anyone can act on. PCA at least offers loadings — the weights of the original features in
each component, which sometimes name themselves, as when the leading component of genotype data
turns out to be geography. Non-linear methods offer nothing comparable. When the deliverable is an
explanation rather than a prediction, [feature selection](https://howaiworks.ai/glossary/feature-selection) may be the
better tool even though it preserves less information, because its output is a list of things that
already have names.
## Code Example
This is the distance-concentration measurement from above, in about ten lines of NumPy. It is
worth running rather than reading: the collapse between 2 and 100 dimensions is the single fact
that motivates everything else on this page.
```python
import numpy as np
rng = np.random.default_rng(0)
for d in (2, 10, 100, 1000):
near, far = [], []
for _ in range(100): # 100 independent query points
X = rng.random((1000, d)) # 1,000 points, uniform in the unit cube
q = rng.random(d) # the query
dist = np.sqrt(((X - q) ** 2).sum(1))
near.append(dist.min())
far.append(dist.max())
n, f = np.median(near), np.median(far)
print(f"d={d:5d} nearest {n:6.3f} farthest {f:6.3f} ratio {f / n:7.2f}")
```
Output:
```
d= 2 nearest 0.015 farthest 1.052 ratio 68.84
d= 10 nearest 0.509 farthest 1.921 ratio 3.77
d= 100 nearest 3.328 farthest 4.772 ratio 1.43
d= 1000 nearest 12.166 farthest 13.630 ratio 1.12
```
Note what does *not* change: the number of points is 1,000 in every row. The data did not get
sparser because there is less of it; it got sparser because the space grew. Adding points barely
helps, either — to keep the same density of 10 samples per axis you would need 10d
points, which is 10 billion at d = 10 and hopeless past that. Reducing *d* is the only lever that
moves, which is the whole argument for doing it.
## Frequently Asked Questions
### Why would you deliberately throw away dimensions?
Because in high dimensions the tools that find structure stop working. With 1,000 points scattered uniformly in 100 dimensions, the farthest point from a query is only about 1.4 times as far away as the nearest one, so nearest-neighbour search, k-means and distance thresholds all lose their meaning. Reduction also cuts memory and compute, but the geometric failure is the reason the field exists.
### Should I use PCA or t-SNE?
PCA if anything downstream consumes the output — clustering, a classifier, a vector index — because it is deterministic, invertible, applies to new points with one matrix multiply, and tells you exactly how much of the data you kept. t-SNE or UMAP only if the output is a picture for a human. Their coordinates have no units and cannot be fed to another algorithm honestly.
### How many components should I keep?
Read it off the cumulative explained-variance curve rather than picking a round number. On MNIST's 60,000 training images, 87 of the 784 possible components carry 90% of the variance, 154 carry 95% and 331 carry 99%. On Fisher's iris data two components carry 97.8%. The right k is a property of the data, and if a supervised model consumes the output, tune k by cross-validated task performance instead.
### Why do clusters in my t-SNE plot look the size they do?
For no reason you can use. t-SNE adapts its distance scale to local density, so a tight cluster and a diffuse one can be drawn at the same size, and the gap between two blobs is not proportional to how different they are. At low perplexity it will also split pure random noise into convincing-looking clusters. Read the plot as a claim that certain points are neighbours, and nothing more.
### What is the difference between dimensionality reduction and feature selection?
Feature selection keeps a subset of your original columns, so the survivors still mean what they meant — useful when someone has to act on the result. Dimensionality reduction builds new composite coordinates from all of them, which usually preserves more information per dimension but produces axes nobody can name.
### Does dimensionality reduction always lose information?
Yes, unless the data genuinely occupies fewer dimensions than it is stored in, which real data often nearly does. With PCA the loss is measurable: the reconstruction error equals exactly the variance in the components you dropped, so keeping 95% of the variance means you can state what the remaining 5% cost you.
## Related
### Related terms
- [Autoencoder](https://howaiworks.ai/glossary/autoencoder)
- [Embedding](https://howaiworks.ai/glossary/embedding)
- [Clustering](https://howaiworks.ai/glossary/clustering)
- [Feature Selection (FS)](https://howaiworks.ai/glossary/feature-selection)
- [Vector Search](https://howaiworks.ai/glossary/vector-search)
- [Unsupervised Learning](https://howaiworks.ai/glossary/unsupervised-learning)
---
Source: https://howaiworks.ai/glossary/dimensionality-reduction — HowAIWorks.ai