Loss Function

Definition

A loss function is the single number a model is trained to make smaller. Give it one prediction and the correct answer and it returns a score for how wrong that prediction was; training is nothing more than repeatedly nudging every parameter in the direction that makes the average of those scores go down.

The number itself is almost incidental. What the training loop consumes is its slope — how much the loss would change if a given weight changed slightly. Backpropagation computes that slope for every parameter in the network and gradient descent steps each one against it. The loss is not a scoreboard the model reads at the end of the run. It is the surface whose steepness moves every weight on every step, which is why a loss with a different shape produces a different model from identical data.

Here is that claim made concrete. Take five true values — 10, 11, 12, 13 and 54 — and ask a model that can only output one constant number what it should say. Under mean squared error the best it can do is 20. Under mean absolute error the best it can do is 12. Under a 0.9-quantile (pinball) loss the best it can do is 54. Same data, same model, answers ranging from 12 to 54, and the only thing that changed was the function being minimised. That is not a quirk of this dataset: the minimiser of squared error is always the mean, the minimiser of absolute error is always the median, and the minimiser of a quantile loss is always that quantile. You are not choosing a way to measure the model. You are choosing what it will become.

How It Works

Each training step runs three things in order. The model produces a prediction, the loss function turns that prediction and the true label into one number, and then the derivative of that number with respect to the prediction is passed backwards through the network by the chain rule, arriving at each weight as a small instruction: increase me, or decrease me, by this much.

The useful way to picture the middle step is that every example in the batch exerts a pull on the prediction, and the loss function decides how strong each pull is. Sitting at the median of the five values above, with the prediction fixed at 12, the pulls look like this:

Squared error's derivative is proportional to the error, so the outlier at 54 pulls with a force of 16.80 against 0.80 from the nearest ordinary point — twenty-one times harder — and the four sensible values are simply outvoted. The prediction slides from 12 to 20 and stays there. Absolute error's derivative is a constant ±1 regardless of distance, so the outlier gets exactly one vote, the same as the point at 13, and the pulls cancel at 12. Huber's derivative grows like MSE's until the error reaches δ and is capped at ±δ beyond it, which is why 54 contributes 0.40 rather than 16.80.

Three consequences follow from this picture. First, the loss must be differentiable in the model's outputs, which is why you cannot train on accuracy: change a weight slightly and the number of correct predictions almost never changes at all, so the gradient is zero nearly everywhere and undefined at the jumps. Cross-entropy exists as the smooth substitute that does move whenever the predicted probability moves. Second, how you reduce the per-example losses into one number is not cosmetic — summing instead of averaging over a batch of 64 multiplies every gradient by 64, which is a silent 64× change to the effective learning rate. Third, anything you add to the loss is something the model will now trade against accuracy: an L2 regularization penalty makes the objective deliberately different from the thing you care about, in exchange for smaller weights and less overfitting.

Types

There is a real taxonomy here, organised by what the model is being asked to produce. What matters in each case is the shape of the gradient it hands back.

Regression losses

Mean squared error gives a gradient proportional to the error, so a point ten times further away pulls ten times harder. This is the right shape when a large error genuinely is disproportionately expensive and the noise is roughly symmetric and light-tailed. It is the wrong shape the moment the data has heavy tails or mislabeled rows, because a single bad record can own the batch's gradient — as the table above shows, at a 21:1 ratio.

Mean absolute error gives every example an equal vote and so ignores outliers by construction. The cost is that the gradient never shrinks as you approach the answer: within a hair of the optimum it is still ±1, so training oscillates around the median unless the learning rate is decayed, and the derivative does not exist at zero error at all.

Huber loss is quadratic inside a band of half-width δ and linear outside it, so the gradient grows with the error up to δ and then flattens. It buys MSE's well-behaved endgame and MAE's outlier immunity for the price of one hyperparameter — and δ is measured in the units of your target, so it silently needs re-tuning every time you rescale the data.

Quantile (pinball) loss charges τ per unit of under-prediction and 1−τ per unit of over-prediction, which makes the gradient deliberately lopsided. Its minimiser is the τ-th quantile, which is how you encode "running out of stock costs us nine times what over-ordering costs". Its wrongness is the mirror image: if your costs really are symmetric, you have introduced a bias for nothing.

Classification losses

Cross-entropy compares two probability distributions, and its gradient at the output logit is exactly p − y — the predicted probability minus the true label. That is an elegantly self-scaling shape: a confidently wrong prediction returns a near-maximal push, a confidently right one returns almost nothing. It is also unbounded, since −log p goes to infinity as p goes to zero, so one confidently mislabeled example produces an arbitrarily large gradient forever. Binary cross-entropy is the two-class case used with logistic regression and for multi-label problems; the categorical form is for mutually exclusive classes.

Focal loss multiplies cross-entropy by (1 − p_t)^γ, where p_t is the probability assigned to the correct class. With γ = 2, an example already scored at p_t = 0.9 contributes 100× less loss than under plain cross-entropy, and one at p_t ≈ 0.968 contributes 1000× less, while a misclassified example is discounted by at most 4× (Lin et al., 2017). The shape it gives the gradient is "spend the update budget on what you are still getting wrong". It is the wrong shape when your classes are roughly balanced and your labels are noisy, because the examples it focuses on are then disproportionately the mislabeled ones.

Hinge loss, used by support vector machines, returns exactly zero gradient once an example is correct by more than the margin — the loss stops caring about examples it has already won. That is what produces a max-margin separator, and it is the wrong shape whenever you need calibrated probabilities, because hinge loss never produces any. The focal loss authors report trying a hinge-style loss on p_t for dense detection and finding it unstable.

Contrastive losses

A contrastive loss has no labels at all. Given a batch of N matched pairs, it forms all N² similarity scores and applies cross-entropy along both axes, so the gradient pulls each true pair together and pushes it away from the N²−N mismatched pairs in the same batch. The difficulty of the task therefore scales with batch size, which is why CLIP was trained with a minibatch of 32,768 and a learnable temperature initialised at the equivalent of 0.07 (Radford et al., 2021). It is the wrong shape when the batch is small — the negatives become too easy and the gradient starves — or when the batch contains near-duplicates, in which case the loss is actively shoving apart two things that ought to be close.

Preference and reward objectives

When the target is "which of these two answers does a person prefer", there is no correct value to subtract from. A reward model is instead fitted with a Bradley-Terry pairwise loss, and Direct Preference Optimization folds that step into a single objective on the policy itself: −log σ(β · [log π(y_w)/π_ref(y_w) − log π(y_l)/π_ref(y_l)]), with β = 0.1 by default. Its gradient raises the likelihood of the preferred completion and lowers the dispreferred one, weighted by how badly the model's own implicit reward currently orders the pair (Rafailov et al., 2023). The shape is purely comparative, and that is exactly its limitation: nothing in the objective pins absolute quality, so a model can satisfy every preference in the dataset while getting worse in ways nobody compared. β controls how far the policy may drift from its reference — too small and it degenerates, too large and it barely moves.

Real-World Applications

RetinaNet and focal loss. Dense object detectors score roughly 100,000 candidate locations per image against a foreground-to-background ratio near 1:1000, and under plain cross-entropy the easy background wins the gradient by sheer weight of numbers. Take a detector that already scores 1000 background locations at p = 0.99 and one real object at p = 0.30: the background owns 89.3% of the total loss. Apply the γ = 2 modulating factor and the background's share falls to 0.2%. In the published ablations, an α-balanced cross-entropy RetinaNet tops out at 31.1 AP on COCO while the identical network trained with focal loss at γ = 2, α = 0.25 reaches 34.0 AP — a 2.9 AP gain from changing nothing but the loss — and 36.0 AP against 32.8 AP for the best online hard example mining baseline on a ResNet-101 backbone (Lin et al., 2017).

CLIP. OpenAI's image-text model is the cleanest published measurement of a loss changing the economics of training. Starting from a bag-of-words baseline, swapping the predictive objective for a contrastive one produced a further 4× improvement in the rate of zero-shot transfer to ImageNet; the transformer language-modelling objective they began with had itself been 3× slower than that same baseline (Radford et al., 2021). Compounded, that is a factor of 12 in training efficiency between the objective they started with and the one they shipped.

The original Transformer. Vaswani et al. trained their WMT 2014 English-to-German system with label smoothing of 0.1 and reported the result plainly: it "hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score" (Vaswani et al., 2017). A deliberate decision to make the training loss worse in order to make the shipped metric better — the gap between loss and goal, stated in one sentence by the people who found it.

Language model post-training. Every instruction-following LLM that has been aligned to human preferences has had its behaviour set by a preference loss, whether through RLHF with a separate reward model or directly through DPO. The "personality" of a chat assistant is not a separate feature; it is the argmin of an objective somebody wrote down.

Key Concepts

Loss, cost and objective are used interchangeably in practice, but when precision matters, loss is the error on one example, cost is the average over a batch or dataset, and objective is what is actually minimised — cost plus regularization penalties and any auxiliary terms.

Convexity means the loss surface has exactly one basin, so any downhill path reaches the global minimum and the optimization problem is solved once the gradient is zero. Squared error over a linear model is convex. The loss surface of a deep network is not, which is why "the loss stopped decreasing" and "the model is as good as it can get" are different statements for a neural network and the same statement for a linear regression.

Multi-task weighting is a hidden hyperparameter hiding inside a plus sign. If a detector's box-regression loss sits around 10 and its classification loss around 0.1, then total = box + cls is a box loss with a rounding error attached, and the weight you did not write down is 100:1.

Loss scaling is a numerical-precision trick, not a modelling one. In FP16 training, small gradients underflow to zero, so implementations multiply the loss by a large constant before backpropagation and divide it out afterwards — an operation performed on the loss purely to keep the gradient representable.

Challenges

Mean squared error on a classification head produces flat gradients exactly where you need them. This is the single most common way a first model silently fails to train. With a sigmoid output and a true label of 1, cross-entropy's gradient at the logit is p − 1, while MSE's is 2(p − 1)·p(1 − p) — the extra p(1 − p) factor is the sigmoid's own derivative, and it collapses toward zero at both ends of the range. At p = 0.01 the two gradients are 0.9900 and 0.0196 in magnitude: MSE's is 50 times weaker. At p = 0.001 they are 0.9990 and 0.0020, a factor of 500. The more confidently wrong the model is, the less MSE tells it to move. The symptom is a training curve that barely descends for many epochs, which looks exactly like a bad learning rate and is not one.

Class imbalance lets a loss report success while the model has learned nothing. On data that is 99% negative, a model that predicts the majority class unconditionally achieves 99% accuracy and a genuinely low average loss. Nothing in the training curve will tell you. The fix is in the objective — class weights, focal loss, or a sampling scheme — and not a bigger network, because the network is already succeeding at the task you specified.

The loss is a proxy, and the gap between the proxy and the goal is where models go wrong in production. Almost nothing anyone is actually paid for is differentiable: revenue, click-through, ranking quality, clinician agreement, whether the unit tests pass. So you minimise something smooth and hope it correlates. Sometimes the correlation is deliberately broken, as with label smoothing above. Sometimes it breaks silently, as when a symmetric regression loss is used for a demand forecast where a stock-out costs nine times what excess inventory costs — the fix there is a quantile loss at τ = 0.9, which in the five-value example from the Definition moves the prediction all the way from 12 to 54.

Unbounded losses make noisy labels expensive. Because −log p diverges, a confidently mislabeled training example generates a very large gradient that never decays, and the model will distort itself to accommodate it — the classification analogue of the outlier at 54 dragging MSE's prediction to 20. Bounded and robust variants exist precisely for datasets where a few percent of the labels are known to be wrong.

A loss that is already at its floor cannot be improved. If 5% of your labels are random, cross-entropy has an irreducible floor set by that noise, and further training past it is memorisation, not learning. A loss that has stopped falling is ambiguous — converged, floored, or starved of gradient — and the three have different fixes.

Future Trends

Non-differentiable objectives are becoming the important ones. Reasoning models are increasingly trained against verifiers: did the unit test pass, does the final answer match, did the proof check. These are exactly the objectives that backpropagation cannot touch, so the "loss" reaches the weights through a policy-gradient estimator instead of the chain rule. The design question shifts from what function to write down to how to get a usable gradient estimate from a binary signal.

Process supervision instead of outcome supervision. Scoring every intermediate reasoning step rather than only the final answer changes what the gradient rewards — a model can no longer be reinforced for reaching the right answer by a wrong route. It also costs far more to label, which is the trade currently being negotiated.

Preference objectives are still being repaired in public. DPO removed the separate reward model from the alignment pipeline, and the failure modes of the resulting loss are now being patched in the objective rather than the model. One is structural: only the ordering of two answers appears in the loss, so nothing constrains absolute quality. Another is length — an unregularized DPO model exploits the mild human preference for longer answers, and adding a length regulariser to the objective recovers close to 20% in win rate on the Anthropic helpful-harmless data at matched output length (Park et al., 2024).

Learned and searched loss functions remain research rather than practice. The idea of treating the loss as something to optimise over, rather than choose, is appealing precisely because the choice is currently made by convention; nothing yet beats a well-matched hand-picked loss reliably enough to displace it.

Code Example

Every arithmetical result quoted above — the three minimisers, the pull table, the gradient comparison and the focal-loss flip — comes from this script, which uses only the standard library. Run it and check.

import math

y = [10.0, 11.0, 12.0, 13.0, 54.0]      # four ordinary values, one outlier

def mse(c): return sum((c - t) ** 2 for t in y) / len(y)
def mae(c): return sum(abs(c - t) for t in y) / len(y)
def pinball(c, tau=0.9):
    return sum(tau * (t - c) if t >= c else (1 - tau) * (c - t) for t in y) / len(y)

# The best single number each loss can pick, by brute force over a fine grid.
grid = [i / 100 for i in range(0, 6001)]
for name, loss in (("MSE", mse), ("MAE", mae), ("pinball(0.9)", pinball)):
    best = min(grid, key=loss)
    print(f"{name:>12}  best constant {best:6.2f}   loss there {loss(best):8.3f}")

# What each point contributes to the gradient at c = 12 (the median).
print()
print(f"{'target':>7}{'MSE pull':>10}{'MAE pull':>10}{'Huber d=2':>11}")
for t in y:
    e = 12.0 - t
    sign = (e > 0) - (e < 0)
    huber = e if abs(e) <= 2 else 2 * sign
    print(f"{t:>7.0f}{2 * e / len(y):>10.2f}{sign / len(y):>10.2f}{huber / len(y):>11.2f}")

# A confidently wrong binary classifier: the true label is 1, the model says p.
# z is the pre-sigmoid logit, so dBCE/dz = p - y and dMSE/dz = 2(p - y)p(1 - p).
print()
print(f"{'p':>7}{'BCE':>9}{'MSE':>9}{'dBCE/dz':>10}{'dMSE/dz':>10}")
for p in (0.5, 0.1, 0.01, 0.001):
    print(f"{p:>7}{-math.log(p):>9.3f}{(1 - p) ** 2:>9.4f}{p - 1:>10.4f}{2 * (p - 1) * p * (1 - p):>10.5f}")

# 1000 easy background locations scored p=0.99, one hard object scored p=0.30.
ce_bg, ce_obj = 1000 * -math.log(0.99), -math.log(0.30)
fl_bg = 1000 * (1 - 0.99) ** 2 * -math.log(0.99)
fl_obj = (1 - 0.30) ** 2 * -math.log(0.30)
print()
print(f"cross-entropy: background owns {ce_bg / (ce_bg + ce_obj):.1%} of the loss")
print(f"focal, g = 2:  background owns {fl_bg / (fl_bg + fl_obj):.1%} of the loss")

Its output:

         MSE  best constant  20.00   loss there  290.000
         MAE  best constant  12.00   loss there    9.200
pinball(0.9)  best constant  54.00   loss there    3.400

 target  MSE pull  MAE pull  Huber d=2
     10      0.80      0.20       0.40
     11      0.40      0.20       0.20
     12      0.00      0.00       0.00
     13     -0.40     -0.20      -0.20
     54    -16.80     -0.20      -0.40

      p      BCE      MSE   dBCE/dz   dMSE/dz
    0.5    0.693   0.2500   -0.5000  -0.25000
    0.1    2.303   0.8100   -0.9000  -0.16200
   0.01    4.605   0.9801   -0.9900  -0.01960
  0.001    6.908   0.9980   -0.9990  -0.00200

cross-entropy: background owns 89.3% of the loss
focal, g = 2:  background owns 0.2% of the loss

The last two lines are the whole argument of this page in miniature. Nothing about the detector changed — not the weights, not the data, not the predictions. Only the function measuring them changed, and the share of the loss devoted to the object the detector exists to find went from 10.7% to 99.8%.