Definition
A support vector machine (SVM) is a classification algorithm that separates two classes using the decision boundary with the widest possible margin — the straight line (or, in higher dimensions, the hyperplane) that sits as far as it can from the nearest point of either class. Those nearest points are called the support vectors, and they are the only points that matter: you could delete every other example in the training set and the boundary would not move.
The reason to want the widest gap is generalization. Many lines separate two clumps of points, but a boundary crammed up against one class will misclassify the first new point that lands just over it, while the line that keeps the largest cushion on both sides has the most room to be right about data it has never seen. SVMs make that intuition precise — they solve for the single maximum-margin boundary — and through the kernel trick they extend it to data that no straight line could separate at all. Introduced by Corinna Cortes and Vladimir Vapnik in their 1995 paper "Support-Vector Networks" (Machine Learning, vol. 20), the SVM was the dominant general-purpose classifier for roughly a decade before deep learning, and it remains a strong choice for supervised learning on small-to-medium datasets.
How It Works
A linear SVM describes its boundary with a weight vector w and an offset b, so a point x lies on the boundary when w·x + b = 0 and falls on one side or the other according to the sign. If we scale things so the closest points of each class satisfy w·x + b = ±1, the margin — the full width of the empty corridor between the classes — works out to exactly 2/‖w‖. Making that corridor as wide as possible therefore means making ‖w‖ as small as possible, subject to every point staying on its correct side. This is a convex optimization with a single global answer, which is part of why SVMs were trusted: unlike a neural network, there is no local minimum to get stuck in.
The solution depends on only a handful of points. In the run at the bottom of this page, a linear SVM on 200 points keeps just 5 of them as support vectors — the ones lying on the margin's edge — and ignores the other 195 entirely. That sparsity is the model: prediction compares a new point only against the support vectors, so a boundary learned from thousands of examples can be stored as a few dozen.
Real data is rarely perfectly separable, so SVMs use a soft margin. A slack term lets some points sit inside the margin, or even on the wrong side, and a parameter C sets the price of doing so. A large C punishes every violation harshly, pulling the boundary tight around the data and risking overfitting; a small C tolerates slack in exchange for a wider, smoother margin that generalizes better. In the run below, dropping C from 10 to 0.01 more than doubles the margin width, from 1.539 to 3.843, and pulls 40 points into the support set instead of 5 — a visibly more relaxed boundary.
The kernel trick
The one genuinely surprising idea in SVMs is how they separate data that no straight line can — points of one class ringed by the other, say. The optimization only ever touches the training points through dot products between pairs of them, x·y, a single number measuring how aligned two points are. The kernel trick replaces that dot product with a kernel function k(x, y) that returns the same alignment the points would have after being lifted into a much higher-dimensional space — without ever computing their coordinates there. A boundary that has to be curved in two dimensions can be a flat hyperplane in that implicit space, and the flat separator maps back down to exactly the curve you wanted.
The most common choice is the radial basis function (RBF) kernel, k(x, y) = exp(−γ‖x − y‖²), which scores a pair by closeness: identical points give 1, and the score falls off smoothly with distance at a rate set by γ. Two points a Euclidean distance of 2 apart, with γ = 0.5, score exp(−0.5 × 2²) = exp(−2) ≈ 0.135. The implicit space this corresponds to is infinite-dimensional, yet each kernel value costs one exponential of a distance — the whole point of the trick is that you buy the expressive power of that space for the price of an ordinary distance calculation.
Types
The word "SVM" spans two distinctions worth knowing. The first is linear versus kernel. A linear SVM uses no kernel; it is fast, its weight vector is directly interpretable, and it is still a strong baseline for high-dimensional problems like text, where the data is often already close to linearly separable. A kernel SVM (RBF, polynomial) trades that speed and interpretability for curved boundaries. The second is classification versus regression: the same margin machinery becomes support vector regression (SVR), which fits a function by keeping points within an ε-wide tube rather than on the correct side of a boundary. In scikit-learn these are SVC and SVR. Multi-class problems are handled by combining several binary SVMs (one-vs-one or one-vs-rest), since the core method draws a boundary between exactly two classes.
Real-World Applications
For roughly a decade before deep learning took over, the SVM was the default whenever a strong classifier was needed and the data was moderate in size. Cortes and Vapnik's original 1995 paper demonstrated the method on the US Postal Service's handwritten-digit images, and digit and character recognition stayed a signature SVM benchmark for years afterward. In text — where a document becomes a very high-dimensional but sparse bag-of-words vector, exactly the regime where a linear SVM shines — SVMs became the standard tool for tasks like spam filtering and topic categorization following Thorsten Joachims' 1998 study of text categorization with SVMs. In bioinformatics they took hold for problems with far more features than samples, such as classifying tissue from gene-expression microarrays, where the margin's resistance to overfitting in high dimensions is a real advantage. The through-line is small-to-medium datasets with many features: that is where SVMs earned their reputation as a machine learning workhorse, and where they remain a sensible first model today.
Challenges
The reason SVMs faded for large-scale problems is in their training cost. Fitting the margin is a quadratic-programming problem whose cost grows roughly between quadratic and cubic in the number of training examples — so a dataset ten times larger can take on the order of a hundred times longer to fit, and millions of examples become impractical where a neural network trained by minibatch gradient descent scales close to linearly. The kernel matrix compounds this: it holds one entry per pair of points, so its memory alone grows with the square of the sample count.
SVMs are also sensitive to feature scaling in a way tree-based models are not. Because the RBF kernel measures raw Euclidean distance, a feature ranging over thousands will drown out one ranging over fractions, and the boundary will effectively ignore the smaller feature. Standardizing every feature to a comparable scale is a correctness requirement, not optional preprocessing — skip it and the model can quietly collapse to using a single column. Finally, a kernel SVM's results are hard to interpret: the boundary lives in an implicit space you never see, so unlike a decision tree there is no readable rule to inspect. Together — poor scaling, mandatory preprocessing, and opaque non-linear boundaries — these are why SVMs gave ground to random forests and neural networks as datasets grew.
Code Example
This short scikit-learn example trains a linear SVM on two blobs of points, reads off the margin width and support-vector count, then softens the margin to show C's effect directly:
import numpy as np
from sklearn.svm import SVC
from sklearn.datasets import make_blobs
# Two clearly separated classes in 2D
X, y = make_blobs(n_samples=200, centers=2, cluster_std=1.2, random_state=6)
# A linear SVM: find the widest-margin separating line
clf = SVC(kernel="linear", C=10)
clf.fit(X, y)
w = clf.coef_[0] # the normal vector to the boundary
margin = 2 / np.linalg.norm(w) # full width of the margin
print(f"margin width : {margin:.3f}")
print(f"support vectors : {len(clf.support_)} of {len(X)} points")
# Soften the margin: a small C tolerates more slack, widening the margin
soft = SVC(kernel="linear", C=0.01).fit(X, y)
print(f"margin width at C=0.01 : {2 / np.linalg.norm(soft.coef_[0]):.3f}")
print(f"support vectors at C=0.01: {len(soft.support_)} of {len(X)} points")
Output:
margin width : 1.539
support vectors : 5 of 200 points
margin width at C=0.01 : 3.843
support vectors at C=0.01: 40 of 200 points
The numbers make the mechanism concrete: at C=10 the boundary rests on just 5 of the 200 points, so the model is sparse in exactly the way the theory promises; loosening to C=0.01 widens the corridor from 1.539 to 3.843 and, because a wider corridor catches more points inside it, grows the support set eightfold. Swapping kernel="linear" for kernel="rbf" would let the same call fit a curved boundary through the kernel trick, at the cost of a gamma to tune and a weight vector you can no longer read directly.