Machine Learning (ML)

Definition

Ordinary programming and machine learning differ in one place: who writes the rule. In ordinary programming you work out the rule yourself — charge $3 plus $0.50 per kilometre — and type it in; the machine only executes it. In machine learning you supply examples of inputs paired with the right answers, plus a way of scoring how wrong any candidate rule is, and a search procedure finds the rule. Nobody states it, and usually nobody could have.

Here is the whole idea at its smallest possible size. Four deliveries, distance in kilometres against time in minutes: (1, 12), (2, 17), (3, 25), (4, 26). Fix the shape of the rule as minutes = w × km + b, leave w and b blank, and let a program hunt for them. It starts at w = 0, b = 0 — a rule that predicts zero minutes for every delivery, average squared error 433.5 — and after three small corrections it sits at w = 6.66, b = 2.52, error 6.38. Left running, it settles on w = 5, b = 7.5: the best straight line those four points admit, with an error of 2.25 that no line can beat.

Nobody wrote the 5. Nobody wrote the 7.5. They were found, by a procedure that at every moment knew only how wrong it currently was and which direction made it less wrong. Every machine learning system is that loop at a different scale — a large language model is this same arithmetic with hundreds of billions of ws instead of one, which is why the worked example below is not a toy but the mechanism itself.

Three things break for readers who take that loosely, and each is worked with numbers further down:

The rule is not written anywhere you can read. Here it is two numbers and you can say "five minutes per kilometre"; at a billion parameters there is no sentence to extract, and a system whose logic a human can audit line by line is a different technology — that is symbolic AI, and it is not what fits data.

The model learns whatever correlates, not what causes. A rule-based model trained on real hospital records learned that a history of asthma lowers your risk of dying from pneumonia. The correlation was real. The cause was that asthmatic patients get sent straight to intensive care.

Accuracy is not usefulness. On email where 2% is spam, a filter that catches nothing scores 98.0%, and a genuinely good one scores 98.5%. Half a percentage point separates useless from valuable, which is why nobody who ships classifiers quotes accuracy alone.

How It Works

Every supervised machine learning system, from that line to a frontier model, is three parts and nothing else.

A model is a shape with adjustable numbers in it. Here the shape is a straight line and the adjustable numbers are w and b. In a neural network the shape is layers of weighted sums with a squashing function between them, and the adjustable numbers — the parameters — run into the billions. You choose the shape. You never choose the numbers.

A loss function collapses "how wrong is this rule on my data" into a single number, so that two candidate rules can be compared. The standard choice for predicting a quantity is mean squared error: take each prediction minus the true value, square it so that overshooting and undershooting both count as error, and average.

An optimiser changes the numbers to make the loss smaller. Gradient descent is the workhorse: compute which way each parameter would have to move to reduce the loss, take a small step that way, repeat.

The loss, computed

At w = 0, b = 0 the model predicts 0 minutes for all four deliveries, so the errors are the true values themselves:

MSE = (12² + 17² + 25² + 26²) / 4
    = (144 + 289 + 625 + 676) / 4
    = 1734 / 4
    = 433.5

That number is the entire feedback signal. The optimiser is not told that distance matters, that time is positive, or that the relationship is roughly linear. It is told 433.5, and that this should go down.

Which way is downhill

Write each error as rᵢ = (w·xᵢ + b) − yᵢ, the prediction minus the truth. Differentiating the mean squared error with respect to each parameter gives two sums:

∂L/∂w = (2/4) × Σ xᵢ·rᵢ
∂L/∂b = (2/4) × Σ rᵢ

At the starting point the errors are (−12, −17, −25, −26), so:

Σ xᵢ·rᵢ = 1(−12) + 2(−17) + 3(−25) + 4(−26) = −225   →   ∂L/∂w = −112.5
Σ rᵢ    = −12 − 17 − 25 − 26                 = −80    →   ∂L/∂b = −40

Both derivatives are negative, meaning "increase me". Multiply by a learning rate of 0.05 and subtract, and one step lands on w = 5.625, b = 2.0. Recompute the loss there and it is 18.24. One step, and 96.3% of all the error that could ever be removed is gone.

The line you would draw through those four points by eye is the bottom row, and the procedure gets there without ever being shown it. That is the sentence to keep.

The long tail, and why "standardise your features" exists

Look at the slope. It overshoots — 5.625, then 6.53, then 6.66, sailing past its final value of 5 — and then walks back down over the next 173 steps while the intercept slowly climbs from 2.52 towards 7.5. Three steps close 99.0% of the gap between the starting loss and the best achievable one; the last 1% costs 173 more.

The cause is that w and b are coupled, because the inputs (1, 2, 3, 4 km) sit far from zero: every change to the slope swings the whole line up or down, which the intercept then has to undo. Subtract the mean distance from each input first, and the identical run at the identical learning rate converges in 47 steps instead of 176 — 3.7× fewer for a one-line change that alters nothing about the data's content. This is why every practical guide opens by telling you to scale your features. It is not hygiene. It is the difference between a fit that finishes and one that crawls, and at a billion parameters it is the difference between a training run and a wasted cluster.

Fitting is not learning

The four points are now fitted. That is not the same as having learned anything, and the distinction is the one most beginners skip.

A model is only useful if it is right about data it has never seen — generalization. Fitting the training data is trivially easy: a cubic curve passes through all four of those delivery points exactly, error 0.0, and it is a far worse predictor than the line, because it has bent itself around noise. That failure is overfitting, and you cannot detect it by looking at training error — by construction, training error says the cubic is perfect.

So you hold data back. When LeCun and colleagues at AT&T Bell Labs trained a network to read handwritten postcodes (NeurIPS 1989), they reported 1.1% error on the training set and 3.4% on the test set. Those two numbers, and the 2.3-point gap between them, are what an honest result looks like. A paper reporting only the 1.1% would be telling you how well the model memorised. Cross-validation and regularization exist to manage that gap; the split that produces it is non-negotiable.

Types

The three-way split is real and universally used, and the thing that actually distinguishes them is where the training signal comes from — what plays the role of yᵢ in the loss above.

Supervised learning attaches the correct answer to every example. Someone labelled it: a radiologist marked the tumour, a user clicked "spam", the delivery was timed at 17 minutes. The loss is a direct comparison against that answer, exactly as on this page. The signal is the strongest available and also the most expensive, because a human produced every unit of it. Predicting a category is classification; predicting a quantity is regression.

Unsupervised learning gets no answers at all. The training signal is manufactured from the data's own structure: a clustering algorithm is scored on how tightly its groups hold together, an autoencoder on how accurately it can reconstruct an input after squeezing it through a narrow bottleneck. Nothing external says which grouping is correct, which is both the appeal — no labelling cost — and the difficulty, since "correct" is defined only by the objective you happened to write.

Reinforcement learning gets neither labels nor structure, only a reward that arrives late. An agent acts, the environment responds, and at some point a score appears. Nobody says which move was the good one, so the algorithm has to spread credit backwards across a whole sequence — the credit assignment problem, which is what makes RL harder than the other two and why it dominates game-playing and robot control rather than spreadsheets.

Two hybrids matter enough to name. Self-supervised learning takes unlabelled data and hides part of it — mask a word, predict it from the rest — turning free text into supervised examples with no annotator involved. Every large language model is trained this way, which is the trick that made foundation models economically possible. Semi-supervised learning mixes a small labelled set with a large unlabelled one, and is what you reach for when labels cost money and raw data does not.

Real-World Applications

Reading the post. In the NeurIPS 1989 paper above, LeCun, Boser, Denker and colleagues trained a convolutional network on 9,298 digits cut from real US Mail passing through the Buffalo, New York post office — 7,291 handwritten digits for training, 2,007 for test, plus printed ones. After 30 passes through the data it reached 3.4% test error, and to hit an operational 1% error it rejected 9% of the handwritten digits for a human to read. The whole training run took about three days on a SPARCstation 1. Nobody wrote a rule for what a handwritten 7 looks like; the rule was fitted from 7,291 examples of other people's handwriting.

Deciding who gets admitted to hospital. Caruana and colleagues' KDD 2015 paper trained models on 14,199 pneumonia patients described by 46 features, of whom 10.86% (1,542) died, to predict probability of death so low-risk patients could be treated as outpatients. Logistic regression reached AUC 0.8432; their intelligible GA²M model reached 0.8576. A second model in the same paper predicts 30-day hospital readmission from 3,956 features across 195,901 training patients. These are the decisions machine learning is already making, and the same paper is the source of this page's sharpest warning, below.

Recommendation at scale. The Netflix Prize released more than 100 million dated ratings covering 480,189 users and 17,770 films, and set its Grand Prize at beating Netflix's own Cinematch system — published quiz-set RMSE 0.9514 — by 10%. It was won in 2009 by BellKor's Pragmatic Chaos, whose final blend drew on a pool of 454 separate predictors, as described in Koren's solution paper. That number is itself a lesson: an ensemble of 454 models wins a leaderboard and is not something anyone runs on every page load. Competition accuracy and deployable accuracy are different goods.

Spam filtering, the example that runs through this page, is the everyday case. Modern filters decide spam-or-not with a fitted classifier rather than a hand-written blocklist, because spam adapts and a blocklist does not — retraining on last week's messages is a rule update nobody has to write.

Key Concepts

• Features and labels — the x and the y. Distance in kilometres is a feature; the 17 minutes is a label. A model with the wrong features cannot be rescued by a better algorithm.
• Parameters versus hyperparameters — w and b are parameters: found by the fit, two of them here, hundreds of billions in a frontier large language model. The learning rate 0.05 and the decision to use a straight line are hyperparameters: chosen by you, before any fitting, and never learned.
• Epoch and step — a step is one parameter update; an epoch is one full pass over the training data. The 1989 postcode network converged in 30 epochs, which its authors attributed to how much redundancy real data contains.
• Overfitting versus underfitting — the cubic through all four delivery points has zero training error and is useless; the best flat horizontal line has error 33.5 and is useless for the opposite reason. Useful models live between them.
• The generalization gap — test error minus training error, 3.4 − 1.1 = 2.3 points in the postcode network. This is the number that tells you whether a result will survive contact with reality.
• Inference — using the fitted model, as opposed to fitting it. Training the line took 176 steps; predicting one new delivery is 5 × km + 7.5, one multiply and one add. Almost all of the cost is on one side and almost all of the usage is on the other.

Challenges

98% accuracy, measured against a 98% base rate

Take 10,000 emails of which 200 (2%) are spam. A filter that flags nothing at all is right 9,800 times out of 10,000: 98.0% accurate, and worthless. Now a real filter:

Accuracy is (150 + 9,700) / 10,000 = 98.5%. Half a point above the do-nothing filter, and a completely different product. The numbers that show the difference are precision — 150 / (150 + 100) = 60% of flagged mail really is spam — and recall — 150 / 200 = 75% of spam is caught. Both are terrible if this is a bank's mail server, because those 100 false positives are real messages in the junk folder. This is the single most common way a machine learning result is oversold, and the fix is arithmetic, not judgement: always compare against the base rate.

The model learns the treatment, not the disease

From the same Caruana paper: a rule-based model trained on pneumonia records learned HasAsthma(x) ⇒ LowerRisk(x) — that a history of asthma predicts lower risk of dying from pneumonia. The pattern was genuinely in the data. The reason was that asthmatic patients presenting with pneumonia were routinely admitted straight to intensive care, and that aggressive care worked. A model deployed on that rule sends asthmatics home. The authors found the same shape of problem lurking in the chronic-lung-disease and history-of-chest-pain terms.

Nothing about the fitting procedure is capable of noticing this, because a correlation caused by an intervention is arithmetically identical to a correlation caused by biology. Only a human who knows the domain can catch it — and only if the model is legible enough to read, which is precisely why that paper prefers a slightly less accurate model it can inspect.

Yesterday's distribution

A fitted model assumes tomorrow's data comes from the same process as the training data. Spam authors rewrite their messages, a pandemic changes what shopping looks like, a new courier changes delivery times. Nothing errors. The model keeps returning confident predictions from a world that has moved, and the accuracy decays silently — which is why monitoring is a core part of MLOps rather than an afterthought. The Netflix ratings above stop in December 2005; a model fitted on them describes 2005 taste, permanently.

Leakage: a feature that already contains the answer

If a hospital dataset includes a "discharged to hospice" flag, a mortality model will find it and score beautifully in testing, because the flag is a consequence of the outcome rather than a predictor of it. In production the field is empty at the moment you need the prediction. Leakage looks exactly like success right up until deployment, and it is the most expensive mistake on this list because the validation step you trust is the step that hides it.

There is no rule to read

With two parameters you can say "five minutes per kilometre". With a billion, there is no such sentence — the "rule" is a tensor, and the field of explainable AI exists because that is a genuine loss and not a presentational one. If your application requires that a human can state why a decision was made, that requirement constrains which models you are allowed to use, and it constrains them before you start.

Future Trends

Labels stopped being the bottleneck. For thirty years the binding constraint on supervised learning was that a person had to annotate every example. Self-supervised pretraining broke that: mask part of the input and predict it, and any unlabelled corpus becomes training data. Practitioners now start from a pretrained model and adapt it — fine-tuning, transfer learning, few-shot learning — which means the typical machine learning project in 2026 begins with numbers somebody else found, and adjusts them.

Deep learning still has not taken the spreadsheet. On tabular data — the format most organisations actually hold — Grinsztajn, Oyallon and Varoquaux benchmarked tree-based models against neural networks across 45 datasets with 20,000 compute hours of hyperparameter search per learner, and found tree ensembles remain state of the art on medium-sized data (~10K samples), before even counting their speed advantage. If your data is rows and columns, gradient boosting is still the default, and the gap is about inductive bias rather than fashion.

Model choice is becoming a budgeting exercise. Scaling laws relate parameters, data and compute to expected loss well enough that the question "what accuracy can I buy for this budget" has an approximate answer before training starts. That is a genuine change in practice: the decision moves from architecture search to resource allocation.

The work is moving from training to keeping it working. Fitting is the short part. Detecting that a deployed model has drifted, deciding when to retrain, versioning the data that produced a given set of parameters, and rolling back a bad fit are the parts that consume teams — an engineering discipline that barely existed a decade ago and now has its own name, MLOps.

Code Example

The whole page in twenty lines of plain Python — no libraries, and nowhere is a value for w or b stated. Both are searched for. The final block computes the exact least-squares answer for comparison, which the search is converging on but never told.

# Fit  minutes = w * km + b  to four deliveries by gradient descent.
data = [(1, 12), (2, 17), (3, 25), (4, 26)]   # (km, minutes)
w, b, lr = 0.0, 0.0, 0.05
n = len(data)

def mse(w, b):
    return sum((w * x + b - y) ** 2 for x, y in data) / n

print(f"step 0    w={w:.4f}  b={b:.4f}  loss={mse(w, b):9.4f}")
for step in (1, 2, 3):
    dw = 2 / n * sum(x * (w * x + b - y) for x, y in data)
    db = 2 / n * sum(    (w * x + b - y) for x, y in data)
    w, b = w - lr * dw, b - lr * db
    print(f"step {step}    w={w:.4f}  b={b:.4f}  loss={mse(w, b):9.4f}")

while mse(w, b) > 1.01 * 2.25:          # run on until within 1% of the best possible
    dw = 2 / n * sum(x * (w * x + b - y) for x, y in data)
    db = 2 / n * sum(    (w * x + b - y) for x, y in data)
    w, b = w - lr * dw, b - lr * db
    step += 1
print(f"step {step}  w={w:.4f}  b={b:.4f}  loss={mse(w, b):9.4f}")

mx = sum(x for x, _ in data) / n        # closed-form least squares, for comparison
my = sum(y for _, y in data) / n
ws = sum((x - mx) * (y - my) for x, y in data) / sum((x - mx) ** 2 for x, _ in data)
bs = my - ws * mx
print(f"exact     w={ws:.4f}  b={bs:.4f}  loss={mse(ws, bs):9.4f}")

Output:

step 0    w=0.0000  b=0.0000  loss= 433.5000
step 1    w=5.6250  b=2.0000  loss=  18.2422
step 2    w=6.5312  b=2.3937  loss=   6.8145
step 3    w=6.6594  b=2.5216  loss=   6.3808
step 176  w=5.1243  b=7.1344  loss=   2.2723
exact     w=5.0000  b=7.5000  loss=   2.2500

Swap the two-parameter line for a million-parameter network and every line of this survives: the loss, the derivative, the small step against it, the loop. What changes is that dw is computed by backpropagation instead of by hand, and that you can no longer print the parameters and understand them.