Definition
Training is the process of fitting a model's parameters to data. You repeatedly show the model examples, measure how wrong its predictions are with a single number called the loss, and nudge every one of its internal parameters a little in the direction that makes the loss smaller — over and over, until the loss stops falling. A fresh model is just a pile of random numbers; "training" is the loop that turns those random numbers into ones that do something useful.
The loop has five steps that repeat: forward pass (run an input through the model to get a prediction), loss (compare the prediction to the correct answer and score the error), backward pass (work out, for each parameter, whether nudging it up or down would lower that error), update (move every parameter a small step in the error-lowering direction), and repeat with the next batch of examples. That is the whole of training. Everything else — the choice of model, the size of the dataset, the hardware bill — is detail hung on this loop.
How It Works
Each parameter update needs a direction and a size. The direction comes from the gradient: for every parameter, the gradient answers "if I increase this number slightly, does the loss go up or down, and how steeply?" Computing all of those partial derivatives in one efficient sweep backward through the model is what backpropagation does. Stepping every parameter a little way against its gradient — downhill on the loss — is gradient descent. The overarching activity of minimizing the loss this way is optimization; training is optimization applied to a model and a dataset.
The size of each step is the learning rate, and it is the single knob that most often decides whether training works. Nudge too gently and the loss crawls down so slowly it may never converge in the time you have; nudge too hard and each step overshoots the bottom of the valley, so the loss bounces around or climbs to infinity instead of settling. There is no universal right value — it depends on the model and the data — which is why practitioners try a few and watch the loss curve.
Epoch, batch, and step
Three words travel with training, and each names a different slice of the loop:
- A batch (or mini-batch) is the chunk of examples the model processes before it does one update. You do not usually feed in the entire dataset at once — that is too much to hold in memory and wastes an update the model could have made sooner — nor one example at a time, which makes the gradient estimate too noisy. A batch of 32, 64, or a few hundred is the usual compromise.
- A step (or iteration) is one trip through the loop: one batch in, one parameter update out.
- An epoch is one full pass over the whole dataset.
The arithmetic that connects them is worth doing once, because it is durable and it demystifies training logs. Take a classic dataset like MNIST's 60,000 training images and a batch size of 32. One epoch is 60,000 / 32 = 1,875 steps. Train for 10 epochs and the model has performed 18,750 parameter updates — it has seen every image ten times, and its weights have been nudged 18,750 times. When a training log prints "step 1875/1875, epoch 1", this is the sum it is counting.
Why training costs so much compute
Each forward pass through a model with N parameters costs roughly 2N floating-point operations per token (one multiply and one add per parameter). The backward pass costs about twice that — roughly 4N — because it has to propagate error signals and accumulate a gradient for every parameter. Adding them gives the standard rule of thumb for total training compute, from Kaplan et al.'s 2020 scaling-laws paper:
C ≈ 6ND FLOPs, where N is the number of parameters and D is the number of training tokens.
The 6 is just 2 (forward) + 4 (backward). Worked on real numbers: GPT-3 had N = 175 billion parameters and was trained on D = 300 billion tokens, so its training compute was on the order of 6 × 175×10⁹ × 300×10⁹ ≈ 3.15 × 10²³ FLOPs. That single number is why frontier training runs happen in datacenters and not on laptops: the loop above is cheap, but you run it across trillions of token-passes.
Why training needs far more memory than inference
Running a finished model — inference — holds one copy of the weights and runs only the forward pass. Training holds much more at once: the weights, the gradients (one number per parameter, produced by the backward pass), and the optimizer state. The dominant optimizer, Adam, keeps two extra running averages per parameter, so with weights + gradients + two Adam buffers you are storing four full-size copies of the model during training — about 16 bytes per parameter in 32-bit precision, roughly 16 GB for a one-billion-parameter model just for these buffers. The optimization page works this memory accounting out in detail. The practical consequence: a model that runs comfortably for inference can be several times too large to train on the same hardware.
Training is also distinct from fine-tuning. Training usually means fitting a model from scratch, starting from random parameters and a large dataset. Fine-tuning starts from an already-trained model and continues the same loop on a smaller, task-specific dataset — the mechanism is identical, but the starting point and the data budget are not.
Real-World Applications
The largest application of training today is pretraining large language models. Models in the GPT, Llama, and Gemini families are trained by running the forward/loss/backward/update loop over trillions of tokens of text, with a next-token prediction loss, until the model reliably predicts what comes next. The scaling laws that guide how big to make the model and how much data to use are empirical findings about exactly this loop.
In computer vision, image classifiers like the ResNet family are trained on labeled image datasets — ImageNet's roughly 1.2 million labeled photos is the canonical benchmark — by the same loop, with a loss that penalizes putting probability on the wrong class. In speech, recommendation, fraud detection, and forecasting, the specifics of the model and loss change but the training loop does not: fit parameters to data by repeated, gradient-guided nudges. The one decision that changes across all of these is what counts as "wrong" — the choice of loss function — and everything downstream is the same machinery.
Challenges
The failure modes of training are mostly failures of the loop, and they are concrete:
- Learning rate too high or too low. Too high and the updates overshoot and the loss diverges — the log shows the loss climbing or turning to
NaN. Too low and the loss falls so slowly the run wastes its entire budget without converging. This is the most common reason a training run fails outright. - Training on the test set (data leakage). The whole point of training is a model that works on data it has never seen, so you hold out a test set to measure that. If any test example leaks into the training data — directly, or through a duplicate, or through a feature computed using future information — the reported accuracy is inflated and the model will disappoint in production. The measurement is only honest if the model was never trained on what you measure it with.
- Training too long (overfitting). Keep running the loop and the loss on the training data keeps dropping, but past a point the model stops learning the general pattern and starts memorizing the specific examples — including their noise. Loss on held-out data starts rising even as training loss falls. The fix is to watch a validation set and stop when it stops improving, not when training loss hits zero.
Code Example
A training loop is small enough to write in full. This one fits a linear rule y = 2·x₁ − 3·x₂ + 1 from 800 noisy examples, using mini-batches of 32 — so one epoch is 800 / 32 = 25 steps. Watch the five-step loop inside the inner for, and watch the loss fall from its random start toward zero as the recovered parameters approach the true 2, −3, 1.
import numpy as np
rng = np.random.default_rng(0)
# 800 synthetic examples of a linear rule: y = 2*x1 - 3*x2 + 1
N = 800
X = rng.normal(size=(N, 2))
y = 2 * X[:, 0] - 3 * X[:, 1] + 1 + 0.1 * rng.normal(size=N)
# parameters we are fitting: two weights and a bias, all starting at 0
w = np.zeros(2)
b = 0.0
batch_size = 32
lr = 0.02
steps_per_epoch = N // batch_size # 800 / 32 = 25
print(f"epoch 0: loss={((X @ w + b - y) ** 2).mean():.4f} (random start)")
for epoch in range(1, 9):
idx = rng.permutation(N)
for s in range(steps_per_epoch):
batch = idx[s * batch_size:(s + 1) * batch_size]
xb, yb = X[batch], y[batch]
pred = xb @ w + b # forward pass
err = pred - yb # how wrong we are
grad_w = 2 * xb.T @ err / batch_size # backward pass (gradients)
grad_b = 2 * err.mean()
w -= lr * grad_w # update: one step downhill
b -= lr * grad_b
full_loss = ((X @ w + b - y) ** 2).mean() # loss over the whole set
print(f"epoch {epoch}: loss={full_loss:.4f} w={w.round(2)} b={b:.2f}")
Running it prints the loss curve — the same shape every real training run produces, just with more parameters and more zeros:
epoch 0: loss=13.8143 (random start)
epoch 1: loss=1.8333 w=[ 1.29 -1.9 ] b=0.62
epoch 2: loss=0.2522 w=[ 1.75 -2.59] b=0.85
epoch 3: loss=0.0425 w=[ 1.91 -2.85] b=0.94
epoch 4: loss=0.0148 w=[ 1.97 -2.94] b=0.97
epoch 5: loss=0.0111 w=[ 1.99 -2.97] b=0.99
epoch 6: loss=0.0106 w=[ 2. -2.99] b=0.99
epoch 7: loss=0.0105 w=[ 2. -2.99] b=0.99
epoch 8: loss=0.0105 w=[ 2. -2.99] b=0.99
The loss drops fast at first, when the parameters are far from right and the gradient is steep, then flattens as the model closes in and the nudges get smaller. When the curve goes flat, training has converged: the loss has stopped falling, which is the signal to stop.