---
source: 'https://howaiworks.ai/glossary/machine-learning'
section: glossary
title: Machine Learning (ML)
description: >-
  How a machine finds a rule nobody wrote: a line fitted to four points by hand,
  loss falling from 433.5 to 6.38 in three steps, and why 98% accuracy lies.
tags:
  - Machine Learning
  - supervised learning
  - unsupervised learning
  - Reinforcement Learning
  - training
  - generalization
  - overfitting
  - gradient descent
category: Machine Learning
datePublished: '2025-07-15'
lastUpdated: '2026-07-22'
---

# Machine Learning (ML)

> How a machine finds a rule nobody wrote: a line fitted to four points by hand, loss falling from 433.5 to 6.38 in three steps, and why 98% accuracy lies.

## Definition

Ordinary programming and machine learning differ in one place: **who writes the rule.** In ordinary
programming you work out the rule yourself — *charge $3 plus $0.50 per kilometre* — and type it in;
the machine only executes it. In machine learning you supply examples of inputs paired with the
right answers, plus a way of scoring how wrong any candidate rule is, and a search procedure finds
the rule. Nobody states it, and usually nobody could have.

Here is the whole idea at its smallest possible size. Four deliveries, distance in kilometres
against time in minutes: (1, 12), (2, 17), (3, 25), (4, 26). Fix the *shape* of the rule as
`minutes = w × km + b`, leave `w` and `b` blank, and let a program hunt for them. It starts at
`w = 0, b = 0` — a rule that predicts zero minutes for every delivery, average squared error
**433.5** — and after three small corrections it sits at `w = 6.66, b = 2.52`, error **6.38**. Left
running, it settles on `w = 5, b = 7.5`: the best straight line those four points admit, with an
error of **2.25** that no line can beat.

Nobody wrote the 5. Nobody wrote the 7.5. They were *found*, by a procedure that at every moment
knew only how wrong it currently was and which direction made it less wrong. Every machine learning
system is that loop at a different scale — a [large language model](https://howaiworks.ai/glossary/large-language-model)
is this same arithmetic with hundreds of billions of `w`s instead of one, which is why the
worked example below is not a toy but the mechanism itself.

Three things break for readers who take that loosely, and each is worked with numbers further down:

The rule is **not written anywhere you can read.** Here it is two numbers and you can say "five
minutes per kilometre"; at a billion parameters there is no sentence to extract, and a system whose
logic a human can audit line by line is a different technology — that is
[symbolic AI](https://howaiworks.ai/glossary/symbolic-ai), and it is not what fits data.

The model learns **whatever correlates, not what causes.** A rule-based model trained on real
hospital records learned that a history of asthma *lowers* your risk of dying from pneumonia. The
correlation was real. The cause was that asthmatic patients get sent straight to intensive care.

**Accuracy is not usefulness.** On email where 2% is spam, a filter that catches nothing scores
98.0%, and a genuinely good one scores 98.5%. Half a percentage point separates useless from
valuable, which is why nobody who ships classifiers quotes accuracy alone.

## How It Works

Every supervised machine learning system, from that line to a frontier model, is three parts and
nothing else.

**A model** is a shape with adjustable numbers in it. Here the shape is a straight line and the
adjustable numbers are `w` and `b`. In a [neural network](https://howaiworks.ai/glossary/neural-network) the shape is
layers of weighted sums with a squashing function between them, and the adjustable numbers —
the [parameters](https://howaiworks.ai/glossary/parameters) — run into the billions. You choose the shape. You never
choose the numbers.

**A [loss function](https://howaiworks.ai/glossary/loss-function)** collapses "how wrong is this rule on my data" into a
single number, so that two candidate rules can be compared. The standard choice for predicting a
quantity is mean squared error: take each prediction minus the true value, square it so that
overshooting and undershooting both count as error, and average.

**An optimiser** changes the numbers to make the loss smaller. [Gradient
descent](https://howaiworks.ai/glossary/gradient-descent) is the workhorse: compute which way each parameter would have
to move to reduce the loss, take a small step that way, repeat.

### The loss, computed

At `w = 0, b = 0` the model predicts 0 minutes for all four deliveries, so the errors are the true
values themselves:

```
MSE = (12² + 17² + 25² + 26²) / 4
    = (144 + 289 + 625 + 676) / 4
    = 1734 / 4
    = 433.5
```

That number is the entire feedback signal. The optimiser is not told that distance matters, that
time is positive, or that the relationship is roughly linear. It is told **433.5**, and that this
should go down.

### Which way is downhill

Write each error as `rᵢ = (w·xᵢ + b) − yᵢ`, the prediction minus the truth. Differentiating the
mean squared error with respect to each parameter gives two sums:

```
∂L/∂w = (2/4) × Σ xᵢ·rᵢ
∂L/∂b = (2/4) × Σ rᵢ
```

At the starting point the errors are `(−12, −17, −25, −26)`, so:

```
Σ xᵢ·rᵢ = 1(−12) + 2(−17) + 3(−25) + 4(−26) = −225   →   ∂L/∂w = −112.5
Σ rᵢ    = −12 − 17 − 25 − 26                 = −80    →   ∂L/∂b = −40
```

Both derivatives are negative, meaning "increase me". Multiply by a **learning rate** of 0.05 and
subtract, and one step lands on `w = 5.625, b = 2.0`. Recompute the loss there and it is **18.24**.
One step, and 96.3% of all the error that could ever be removed is gone.

| step | w | b | mean squared error |
|---|---|---|---|
| 0 | 0.0000 | 0.0000 | 433.5000 |
| 1 | 5.6250 | 2.0000 | 18.2422 |
| 2 | 6.5312 | 2.3937 | 6.8145 |
| 3 | 6.6594 | 2.5216 | 6.3808 |
| 176 | 5.1243 | 7.1344 | 2.2723 |
| exact answer | 5.0000 | 7.5000 | 2.2500 |

The line you would draw through those four points by eye is the bottom row, and the procedure gets
there without ever being shown it. That is the sentence to keep.

### The long tail, and why "standardise your features" exists

Look at the slope. It overshoots — 5.625, then 6.53, then 6.66, sailing past its final value of 5 —
and then walks back down over the next 173 steps while the intercept slowly climbs from 2.52 towards
7.5. Three steps close **99.0%** of the gap between the starting loss and the best achievable one;
the last 1% costs 173 more.

The cause is that `w` and `b` are coupled, because the inputs (1, 2, 3, 4 km) sit far from zero:
every change to the slope swings the whole line up or down, which the intercept then has to undo.
Subtract the mean distance from each input first, and the identical run at the identical learning
rate converges in **47 steps instead of 176** — 3.7× fewer for a one-line change that alters
nothing about the data's content. This is why every practical guide opens by telling you to scale
your features. It is not hygiene. It is the difference between a fit that finishes and one that
crawls, and at a billion parameters it is the difference between a training run and a wasted
cluster.

### Fitting is not learning

The four points are now fitted. That is not the same as having *learned* anything, and the
distinction is the one most beginners skip.

A model is only useful if it is right about data it has never seen — [generalization](https://howaiworks.ai/glossary/generalization).
Fitting the [training](https://howaiworks.ai/glossary/training) data is trivially easy: a cubic curve passes through all
four of those delivery points *exactly*, error 0.0, and it is a far worse predictor than the line,
because it has bent itself around noise. That failure is [overfitting](https://howaiworks.ai/glossary/overfitting), and
you cannot detect it by looking at training error — by construction, training error says the cubic
is perfect.

So you hold data back. When LeCun and colleagues at AT&T Bell Labs trained a network to read
handwritten postcodes ([NeurIPS 1989](https://proceedings.neurips.cc/paper_files/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf)),
they reported **1.1% error on the training set and 3.4% on the test set**. Those two numbers, and
the 2.3-point gap between them, are what an honest result looks like. A paper reporting only the
1.1% would be telling you how well the model memorised. [Cross-validation](https://howaiworks.ai/glossary/cross-validation)
and [regularization](https://howaiworks.ai/glossary/regularization) exist to manage that gap; the split that produces it
is non-negotiable.

## Types

The three-way split is real and universally used, and the thing that actually distinguishes them is
**where the training signal comes from** — what plays the role of `yᵢ` in the loss above.

**[Supervised learning](https://howaiworks.ai/glossary/supervised-learning)** attaches the correct answer to every
example. Someone labelled it: a radiologist marked the tumour, a user clicked "spam", the delivery
was timed at 17 minutes. The loss is a direct comparison against that answer, exactly as on this
page. The signal is the strongest available and also the most expensive, because a human produced
every unit of it. Predicting a category is [classification](https://howaiworks.ai/glossary/classification); predicting a
quantity is [regression](https://howaiworks.ai/glossary/regression).

**[Unsupervised learning](https://howaiworks.ai/glossary/unsupervised-learning)** gets no answers at all. The training
signal is manufactured from the data's own structure: a [clustering](https://howaiworks.ai/glossary/clustering) algorithm
is scored on how tightly its groups hold together, an
[autoencoder](https://howaiworks.ai/glossary/autoencoder) on how accurately it can reconstruct an input after
squeezing it through a narrow bottleneck. Nothing external says which grouping is correct, which is
both the appeal — no labelling cost — and the difficulty, since "correct" is defined only by the
objective you happened to write.

**[Reinforcement learning](https://howaiworks.ai/glossary/reinforcement-learning)** gets neither labels nor structure,
only a **reward that arrives late**. An agent acts, the environment responds, and at some point a
score appears. Nobody says which move was the good one, so the algorithm has to spread credit
backwards across a whole sequence — the credit assignment problem, which is what makes RL harder
than the other two and why it dominates game-playing and robot control rather than spreadsheets.

Two hybrids matter enough to name. **[Self-supervised learning](https://howaiworks.ai/glossary/self-supervised-learning)**
takes unlabelled data and hides part of it — mask a word, predict it from the rest — turning free
text into supervised examples with no annotator involved. Every large language model is trained this
way, which is the trick that made [foundation models](https://howaiworks.ai/glossary/foundation-models) economically
possible. **Semi-supervised learning** mixes a small labelled set with a large unlabelled one, and
is what you reach for when labels cost money and raw data does not.

## Real-World Applications

**Reading the post.** In the [NeurIPS 1989 paper](https://proceedings.neurips.cc/paper_files/paper/1989/file/53c3bce66e43be4f209556518c2fcb54-Paper.pdf)
above, LeCun, Boser, Denker and colleagues trained a
[convolutional network](https://howaiworks.ai/glossary/convolutional-neural-network) on 9,298 digits cut from real US
Mail passing through the Buffalo, New York post office — 7,291 handwritten digits for training,
2,007 for test, plus printed ones. After 30 passes through the data it reached 3.4% test error, and
to hit an operational 1% error it rejected 9% of the handwritten digits for a human to read. The
whole training run took about three days on a SPARCstation 1. Nobody wrote a rule for what a
handwritten 7 looks like; the rule was fitted from 7,291 examples of other people's handwriting.

**Deciding who gets admitted to hospital.** Caruana and colleagues' [KDD 2015
paper](https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/KDD2015FinalDraftIntelligibleModels4HealthCare_igt143e-caruanaA.pdf)
trained models on 14,199 pneumonia patients described by 46 features, of whom 10.86% (1,542) died,
to predict probability of death so low-risk patients could be treated as outpatients. Logistic
regression reached AUC 0.8432; their intelligible GA²M model reached 0.8576. A second model in the
same paper predicts 30-day hospital readmission from 3,956 features across 195,901 training
patients. These are the decisions machine learning is already making, and the same paper is the
source of this page's sharpest warning, below.

**Recommendation at scale.** The Netflix Prize released more than 100 million dated ratings covering
480,189 users and 17,770 films, and set its Grand Prize at beating Netflix's own Cinematch system —
published quiz-set RMSE **0.9514** — by 10%. It was won in 2009 by BellKor's Pragmatic
Chaos, whose final blend drew on a pool of **454 separate predictors**, as described in [Koren's
solution paper](https://www.asc.ohio-state.edu/statistics/dmsl/GrandPrize2009_BPC_BellKor.pdf).
That number is itself a lesson: an ensemble of 454 models wins a leaderboard and is not something
anyone runs on every page load. Competition accuracy and deployable accuracy are different goods.

**Spam filtering**, the example that runs through this page, is the everyday case. Modern filters
decide spam-or-not with a fitted classifier rather than a hand-written blocklist, because spam
adapts and a blocklist does not — retraining on last week's messages is a rule update nobody has to
write.

## Key Concepts

- **Features and labels** — the `x` and the `y`. Distance in kilometres is a feature; the 17 minutes
  is a label. A model with the wrong features cannot be rescued by a better algorithm.
- **Parameters versus hyperparameters** — `w` and `b` are parameters: found by the fit, two of them
  here, hundreds of billions in a frontier [large language model](https://howaiworks.ai/glossary/large-language-model).
  The learning rate 0.05 and the decision to use a
  straight line are hyperparameters: chosen by you, before any fitting, and never learned.
- **Epoch and step** — a step is one parameter update; an epoch is one full pass over the training
  data. The 1989 postcode network converged in 30 epochs, which its authors attributed to how much
  redundancy real data contains.
- **[Overfitting](https://howaiworks.ai/glossary/overfitting) versus [underfitting](https://howaiworks.ai/glossary/underfitting)** — the cubic
  through all four delivery points has zero training error and is useless; the best flat horizontal
  line has error 33.5 and is useless for the opposite reason. Useful models live between them.
- **The generalization gap** — test error minus training error, 3.4 − 1.1 = 2.3 points in the postcode
  network. This is the number that tells you whether a result will survive contact with reality.
- **[Inference](https://howaiworks.ai/glossary/inference)** — using the fitted model, as opposed to fitting it. Training the
  line took 176 steps; predicting one new delivery is `5 × km + 7.5`, one multiply and one add. Almost
  all of the cost is on one side and almost all of the usage is on the other.

## Challenges

### 98% accuracy, measured against a 98% base rate

Take 10,000 emails of which 200 (2%) are spam. A filter that flags nothing at all is right 9,800
times out of 10,000: **98.0% accurate**, and worthless. Now a real filter:

|  | predicted spam | predicted legitimate |
|---|---|---|
| **actually spam** (200) | 150 | 50 |
| **actually legitimate** (9,800) | 100 | 9,700 |

Accuracy is (150 + 9,700) / 10,000 = **98.5%**. Half a point above the do-nothing filter, and a
completely different product. The numbers that show the difference are **precision** —
150 / (150 + 100) = **60%** of flagged mail really is spam — and **recall** — 150 / 200 = **75%** of
spam is caught. Both are terrible if this is a bank's mail server, because those 100 false positives
are real messages in the junk folder. This is the single most common way a machine learning result
is oversold, and the fix is arithmetic, not judgement: always compare against the base rate.

### The model learns the treatment, not the disease

From the same Caruana paper: a rule-based model trained on pneumonia records learned
`HasAsthma(x) ⇒ LowerRisk(x)` — that a history of asthma predicts *lower* risk of dying from
pneumonia. The pattern was genuinely in the data. The reason was that asthmatic patients presenting
with pneumonia were routinely admitted straight to intensive care, and that aggressive care worked.
A model deployed on that rule sends asthmatics home. The authors found the same shape of problem
lurking in the chronic-lung-disease and history-of-chest-pain terms.

Nothing about the fitting procedure is capable of noticing this, because a correlation caused by an
intervention is arithmetically identical to a correlation caused by biology. Only a human who knows
the domain can catch it — and only if the model is legible enough to read, which is precisely why
that paper prefers a slightly less accurate model it can inspect.

### Yesterday's distribution

A fitted model assumes tomorrow's data comes from the same process as the training data. Spam
authors rewrite their messages, a pandemic changes what shopping looks like, a new courier changes
delivery times. Nothing errors. The model keeps returning confident predictions from a world that
has moved, and the accuracy decays silently — which is why [monitoring](https://howaiworks.ai/glossary/monitoring) is a
core part of [MLOps](https://howaiworks.ai/glossary/mlops) rather than an afterthought. The Netflix ratings above stop in
December 2005; a model fitted on them describes 2005 taste, permanently.

### Leakage: a feature that already contains the answer

If a hospital dataset includes a "discharged to hospice" flag, a mortality model will find it and
score beautifully in testing, because the flag is a consequence of the outcome rather than a
predictor of it. In production the field is empty at the moment you need the prediction. Leakage
looks exactly like success right up until deployment, and it is the most expensive mistake on this
list because the validation step you trust is the step that hides it.

### There is no rule to read

With two parameters you can say "five minutes per kilometre". With a billion, there is no such
sentence — the "rule" is a tensor, and the field of [explainable AI](https://howaiworks.ai/glossary/explainable-ai)
exists because that is a genuine loss and not a presentational one. If your application requires
that a human can state why a decision was made, that requirement constrains which models you are
allowed to use, and it constrains them before you start.

## Future Trends

**Labels stopped being the bottleneck.** For thirty years the binding constraint on supervised
learning was that a person had to annotate every example. Self-supervised pretraining broke that:
mask part of the input and predict it, and any unlabelled corpus becomes training data.
Practitioners now start from a pretrained model and adapt it — [fine-tuning](https://howaiworks.ai/glossary/fine-tuning),
[transfer learning](https://howaiworks.ai/glossary/transfer-learning), [few-shot learning](https://howaiworks.ai/glossary/few-shot-learning) —
which means the typical machine learning project in 2026 begins with numbers somebody else found,
and adjusts them.

**Deep learning still has not taken the spreadsheet.** On tabular data — the format most
organisations actually hold — [Grinsztajn, Oyallon and Varoquaux](https://arxiv.org/abs/2207.08815)
benchmarked tree-based models against neural networks across 45 datasets with 20,000 compute hours
of hyperparameter search per learner, and found tree ensembles remain state of the art on
medium-sized data (~10K samples), before even counting their speed advantage. If your data is rows
and columns, [gradient boosting](https://howaiworks.ai/glossary/gradient-boosting) is still the default, and the gap is
about inductive bias rather than fashion.

**Model choice is becoming a budgeting exercise.** [Scaling laws](https://howaiworks.ai/glossary/scaling-laws) relate
parameters, data and compute to expected loss well enough that the question "what accuracy can I buy
for this budget" has an approximate answer before training starts. That is a genuine change in
practice: the decision moves from architecture search to resource allocation.

**The work is moving from training to keeping it working.** Fitting is the short part. Detecting
that a deployed model has drifted, deciding when to retrain, versioning the data that produced a
given set of parameters, and rolling back a bad fit are the parts that consume teams — an
engineering discipline that barely existed a decade ago and now has its own name,
[MLOps](https://howaiworks.ai/glossary/mlops).

## Code Example

The whole page in twenty lines of plain Python — no libraries, and nowhere is a value for `w` or `b`
stated. Both are searched for. The final block computes the exact least-squares answer for
comparison, which the search is converging on but never told.

```python
# Fit  minutes = w * km + b  to four deliveries by gradient descent.
data = [(1, 12), (2, 17), (3, 25), (4, 26)]   # (km, minutes)
w, b, lr = 0.0, 0.0, 0.05
n = len(data)

def mse(w, b):
    return sum((w * x + b - y) ** 2 for x, y in data) / n

print(f"step 0    w={w:.4f}  b={b:.4f}  loss={mse(w, b):9.4f}")
for step in (1, 2, 3):
    dw = 2 / n * sum(x * (w * x + b - y) for x, y in data)
    db = 2 / n * sum(    (w * x + b - y) for x, y in data)
    w, b = w - lr * dw, b - lr * db
    print(f"step {step}    w={w:.4f}  b={b:.4f}  loss={mse(w, b):9.4f}")

while mse(w, b) > 1.01 * 2.25:          # run on until within 1% of the best possible
    dw = 2 / n * sum(x * (w * x + b - y) for x, y in data)
    db = 2 / n * sum(    (w * x + b - y) for x, y in data)
    w, b = w - lr * dw, b - lr * db
    step += 1
print(f"step {step}  w={w:.4f}  b={b:.4f}  loss={mse(w, b):9.4f}")

mx = sum(x for x, _ in data) / n        # closed-form least squares, for comparison
my = sum(y for _, y in data) / n
ws = sum((x - mx) * (y - my) for x, y in data) / sum((x - mx) ** 2 for x, _ in data)
bs = my - ws * mx
print(f"exact     w={ws:.4f}  b={bs:.4f}  loss={mse(ws, bs):9.4f}")
```

Output:

```
step 0    w=0.0000  b=0.0000  loss= 433.5000
step 1    w=5.6250  b=2.0000  loss=  18.2422
step 2    w=6.5312  b=2.3937  loss=   6.8145
step 3    w=6.6594  b=2.5216  loss=   6.3808
step 176  w=5.1243  b=7.1344  loss=   2.2723
exact     w=5.0000  b=7.5000  loss=   2.2500
```

Swap the two-parameter line for a million-parameter network and every line of this survives: the
loss, the derivative, the small step against it, the loop. What changes is that `dw` is computed by
[backpropagation](https://howaiworks.ai/glossary/backpropagation) instead of by hand, and that you can no longer print
the parameters and understand them.

## Frequently Asked Questions

### What is machine learning in simple terms?

In ordinary programming you work out the rule and type it in. In machine learning you supply examples of inputs paired with correct answers, plus a way of scoring how wrong a candidate rule is, and a search procedure finds the rule for you. Fit a line to four delivery times and the program lands on 'minutes = 5 × km + 7.5' — nobody wrote the 5 or the 7.5, and nobody could have known them in advance.

### How is machine learning different from regular programming?

Regular programming goes rule plus data produces answers. Machine learning goes data plus answers produces a rule. That reversal is the whole definition, and it costs you something real: the rule you get back is a list of numbers rather than readable logic, so a system with billions of parameters cannot be audited the way an if-statement can.

### Do you need a lot of data for machine learning?

It depends entirely on how many numbers the model has to find. The line on this page has two parameters and four examples. The 1989 US Postal Service digit reader had roughly 10,000 training images. A modern large language model has hundreds of billions of parameters and is trained on trillions of tokens. The rough rule is that data has to grow with parameters, or the model memorises instead of generalising.

### What are the three main types of machine learning?

They are told apart by where the training signal comes from. Supervised learning gets the correct answer attached to every example. Unsupervised learning gets no answers at all and is scored on how well it reconstructs or compresses the data's own structure. Reinforcement learning gets neither, only a delayed reward after a sequence of actions, and has to work out which action earned it.

### Why can a 98% accurate model still be useless?

Because accuracy is measured against the base rate, not against zero. If 2% of email is spam, a filter that marks nothing as spam is 98.0% accurate and catches nothing. A genuinely useful filter in the same inbox scores 98.5% — half a point higher — while catching 75% of spam at the cost of 100 legitimate messages misfiled. Precision and recall separate those two systems; accuracy does not.

### Is machine learning the same as AI?

No. Machine learning is one approach within artificial intelligence, the one where behaviour is fitted from data. The older approach, symbolic AI, encodes rules a human wrote and can still read. Almost everything called AI in 2026 is machine learning, but the two words are not interchangeable and the difference is exactly whether a person could inspect the rule.

## Related

### Related terms

- [Artificial Intelligence (AI)](https://howaiworks.ai/glossary/artificial-intelligence)
- [Supervised Learning](https://howaiworks.ai/glossary/supervised-learning)
- [Unsupervised Learning](https://howaiworks.ai/glossary/unsupervised-learning)
- [Reinforcement Learning (RL)](https://howaiworks.ai/glossary/reinforcement-learning)
- [Deep Learning](https://howaiworks.ai/glossary/deep-learning)
- [Gradient Descent](https://howaiworks.ai/glossary/gradient-descent)

---

Source: https://howaiworks.ai/glossary/machine-learning — HowAIWorks.ai