Principal Component Analysis (PCA)

Definition

Principal Component Analysis (PCA) is a technique for reducing the number of variables in a dataset while discarding as little information as possible. It finds a new set of axes — the principal components — that point along the directions in which the data varies most, then describes each data point by its position along just the first few of those axes instead of all the original features. Because the components are ranked by how much variance they capture, you can drop the last ones and know exactly how much you lost.

The word doing the work here is variance. PCA treats variance as a proxy for information: a direction along which the data is spread out tells you something that distinguishes one point from another, while a direction along which every point sits at nearly the same value tells you almost nothing. In the worked example below, a three-feature dataset is squeezed into two components that together retain 99.79% of the total variance — so one whole column is thrown away at a cost of two-tenths of one percent of the information. That trade is the entire point of the method, and it is measurable rather than a matter of judgement.

PCA is unsupervised — it never sees a target or label — and it is linear: every component is a straight-line combination of the original features. Those two facts are also its two main limitations, and both are covered below.

How It Works

PCA runs in four steps, and none of them is optional.

1. Center the data. Subtract each feature's mean so every column has mean zero. PCA measures variance around the mean, and skipping this step folds the position of the cloud into the first component, corrupting every direction after it.

2. Build the covariance matrix. For data with d features this is a d × d symmetric matrix whose diagonal holds each feature's variance and whose off-diagonal entries hold how each pair of features moves together. This matrix is the complete summary of the linear structure PCA can see.

3. Find its eigenvectors and eigenvalues. The eigenvectors of the covariance matrix are the principal components — the new axes. Each has a corresponding eigenvalue equal to the variance of the data measured along that axis. Because the covariance matrix is symmetric, the eigenvectors are guaranteed orthogonal (mutually perpendicular), which is why the components describe non-overlapping directions of variation.

4. Rank, keep, and project. Sort the eigenvectors by eigenvalue from largest to smallest, keep the top k, and project the centered data onto them. The result has k columns instead of d.

What "variance explained" means, worked by hand

Suppose two centered features have this covariance matrix:

C = | 4  2 |
    | 2  4 |

The eigenvalues solve det(C − λI) = 0, which for a 2×2 matrix is (4 − λ)² − 2² = 0, so 4 − λ = ±2, giving λ = 6 and λ = 2. Notice the eigenvalues sum to 8, exactly the sum of the diagonal (4 + 4) — the total variance is conserved, just redistributed onto the new axes. The variance explained by each component is its eigenvalue divided by that total:

PC1: 6 / (6 + 2) = 0.75   ->  75% of the variance
PC2: 2 / (6 + 2) = 0.25   ->  25% of the variance

So projecting this two-feature data onto its single first component keeps 75% of the total variance and halves the number of columns. The eigenvector for λ = 6 solves (C − 6I)v = 0, i.e. [[−2, 2], [2, −2]] v = 0, which forces v₁ = v₂ — the direction (1, 1)/√2, the 45° diagonal. That makes sense: the two features are positively correlated (off-diagonal +2), so they vary together, and the direction of greatest spread runs diagonally between them.

SVD, the route real libraries take

Forming and eigen-decomposing the covariance matrix is the clearest way to explain PCA, but production code rarely does it that way. The singular value decomposition (SVD) of the centered data matrix gives the same principal components directly and with better numerical stability — squaring the data into a covariance matrix throws away precision. scikit-learn's PCA uses SVD internally. The two routes agree exactly in exact arithmetic: the squared singular values are proportional to the eigenvalues of the covariance matrix.

Real-World Applications

PCA is one of the most widely deployed algorithms in data analysis precisely because it is fast, deterministic, and produces a number you can defend. Specific, well-documented uses include:

• Eigenfaces in face recognition. Turk and Pentland's 1991 "Eigenfaces" work represented face images as combinations of a small number of principal components computed from a set of training faces. A face that originally needed thousands of pixel values could be stored and compared as a few dozen coordinates, making early face recognition tractable on the hardware of the day.

• Population genetics. Applying PCA to hundreds of thousands of genetic markers (SNPs) across individuals recovers population structure with no geographic input at all. Novembre and colleagues' 2008 study famously showed that the first two principal components of European genotype data reproduce a recognizable map of Europe — a result summarized in the paper's title, "Genes mirror geography within Europe."

• Yield-curve analysis in finance. Running PCA on daily changes in interest rates across maturities reliably yields three dominant components that practitioners read as level, slope, and curvature. These three factors together capture the great majority of day-to-day movement in the whole curve, which is why fixed-income desks hedge and stress-test in terms of them rather than in terms of every individual maturity.

• Preprocessing and visualization. PCA is the standard first move for compressing many correlated features before feeding them to a model, and for plotting high-dimensional data in two dimensions to eyeball its structure. sklearn.decomposition.PCA is the reference implementation most of this runs through.

If you are reaching for a glossary-style "PCA is used in many fields" list, note that the cases above share a concrete shape: the raw dimensionality is high, the underlying structure is roughly linear, and a small number of components genuinely explains most of the variance. Where that shape does not hold, PCA is the wrong tool — see below.

Key Concepts

Choosing how many components to keep. Plot the variance explained by each component in descending order — the scree plot — and look for the point where the curve flattens, or keep enough components to reach a cumulative target such as 95%. There is no universally correct k; it is a trade-off between compression and fidelity that the variance-explained numbers let you make explicitly.

Standardize when features are on different scales. PCA maximizes variance, and variance depends on units. If one feature is a salary in dollars (variance in the millions) and another is an age in years (variance in the hundreds), the first principal component will point almost entirely along salary — not because salary is more informative but because its raw numbers are bigger. Standardizing each feature to zero mean and unit variance first (equivalently, running PCA on the correlation matrix instead of the covariance matrix) removes the artifact. This is the single most common PCA mistake.

Loadings tell you what a component means. Each component is a weighted mixture of the original features; those weights are the loadings. Reading them is how you interpret an axis — a first component with large positive loadings on every feature usually means "overall size," for instance. But because components are mixtures, they are rarely as clean to name as an original feature.

PCA is linear; the interesting structure may not be. PCA can only rotate and project — it finds flat directions through the cloud. When the meaningful variation lies on a curved manifold, the linear components miss it. For visualization, t-SNE and UMAP capture nonlinear neighborhood structure far better; for a learned nonlinear reduction, an autoencoder generalizes the idea. Kernel PCA extends the linear method to nonlinear boundaries. PCA remains the fast, deterministic baseline you try first, and often the input those methods run on top of.

Challenges

• High variance is not the same as useful signal. Because PCA is unsupervised, it ranks directions by spread, not by relevance to any task. The direction that best separates two classes can be a low-variance one that PCA drops. Using PCA blindly as feature reduction before a classifier can therefore delete exactly the information the classifier needed — a real and easy-to-miss failure.

• Components are hard to interpret. Every component mixes all the original features, so a stakeholder asking "what is component 2?" gets an answer like "0.4 × income minus 0.3 × age plus …" rather than a single meaningful quantity. PCA buys compression at the cost of interpretability.

• Sensitivity to outliers. Variance is a sum of squared deviations, so a single extreme point can pull a principal component toward itself and distort the whole decomposition. Robust variants exist, but plain PCA assumes the outliers have been handled.

• Sign and rotation are ambiguous. An eigenvector and its negation are equally valid, so the sign of a component is arbitrary and can flip between runs or libraries; do not read meaning into it. When several components share nearly equal eigenvalues, the specific directions within that subspace are effectively arbitrary too.

Code Example

This runs PCA from scratch on a small three-feature dataset, reducing it to two components and reporting the variance kept. The output shown below is the actual output of running this exact block.

import numpy as np

# Six samples, three measured features (all in the same units)
X = np.array([
    [2.0, 1.5, 4.2],
    [3.5, 3.8, 6.1],
    [1.0, 1.4, 2.6],
    [4.2, 3.9, 9.0],
    [2.8, 3.1, 4.8],
    [0.5, 0.2, 1.6],
])

# 1. Center each column on its mean (PCA never skips this step)
Xc = X - X.mean(axis=0)

# 2. Covariance matrix of the centered data (3x3, one row/col per feature)
C = np.cov(Xc, rowvar=False)

# 3. Eigen-decomposition: eigenvectors are the principal components,
#    eigenvalues are the variance captured along each one
vals, vecs = np.linalg.eigh(C)          # ascending order
order = np.argsort(vals)[::-1]          # sort descending
vals, vecs = vals[order], vecs[:, order]

# 4. Variance explained
ratio = vals / vals.sum()

np.set_printoptions(precision=4, suppress=True)
print("eigenvalues:      ", vals)
print("variance ratio:   ", ratio)
print("cumulative:       ", np.cumsum(ratio))

# 5. Project the 6x3 data onto the first two components -> 6x2
Z = Xc @ vecs[:, :2]
print("projected shape:  ", Z.shape)

Output:

eigenvalues:       [10.8929  0.3539  0.0233]
variance ratio:    [0.9665 0.0314 0.0021]
cumulative:        [0.9665 0.9979 1.    ]

The first component alone captures 96.65% of the variance, and the first two together reach 99.79%. Dropping the third dimension therefore costs about 0.21% of the total variance — a concrete example of the trade PCA lets you quantify before you make it. Swapping the manual eigen-decomposition for sklearn.decomposition.PCA(n_components=2) gives the same variance ratios (up to the sign of each component, which is arbitrary).