Backpropagation

Definition

A neural network learns by changing its weights, and to change a weight sensibly you need to know one thing: would raising it make the network's error larger or smaller, and by how much. Backpropagation is the algorithm that answers that question for every weight in the network simultaneously — millions or billions of them — starting from a single number, the loss, by applying the chain rule of calculus backwards through the layers.

Note what it does not do: it changes nothing. Backpropagation produces the gradient — the list of "how much does the loss move if I nudge this one parameter" — and stops. Gradient descent is the separate and much simpler step that takes that list and subtracts a fraction of it from the weights. Backpropagation is the measurement; gradient descent is the move. Conflating them is the most common confusion in the whole subject, and keeping them apart explains why you can switch from SGD to Adam without touching the backward pass at all.

The algorithm was formalized for neural networks in "Learning representations by back-propagating errors" (Rumelhart, Hinton & Williams, 1986). What loss.backward() runs in PyTorch today is the same idea in general form: reverse-mode automatic differentiation.

How It Works

Start with the obvious method, because understanding why it fails is the entire justification for the clever one. To find how much a weight contributed to the loss, nudge it by a tiny amount ε, run the whole network forward again, and see how far the loss moved; divide by ε. This is the finite-difference approximation, and it is correct. It is also, at any real scale, arithmetically hopeless. Each gradient costs one full forward pass — two, if you nudge in both directions for accuracy — and a network does not have one weight. A modest model with 1,000,000 parameters therefore needs 1,000,000 forward passes to produce the gradient for one training step. At one millisecond per pass that is about 17 minutes per step, and a routine 10,000-step run would take roughly four months. The cost scales linearly with parameter count, so a billion-parameter model is a thousand times worse again. The naive approach is not slow; it is impossible.

Backpropagation computes all one million gradients in the cost of one forward pass plus one backward pass. The backward pass runs about twice the arithmetic of the forward pass, because each weight matrix is used twice on the way back — once to produce the gradient flowing down to the layer below, once to produce the gradient of the weights themselves — against once on the way in. Total: roughly 3x a forward pass, independent of how many parameters the network has. This 1:2 ratio is where the standard training-cost estimate of 6ND FLOPs comes from: 2 FLOPs per parameter per token forward, 4 backward, for N parameters over D tokens. Against a million forward passes, a cost of three is a speedup north of 300,000x — and the factor grows with every parameter you add.

The saving comes from not throwing work away. Written out, the chain rule for one weight is dL/dw = dL/da · da/dz · dz/dw, and every weight in a given layer shares almost all of that product with every other weight in the same layer, and with every weight in every layer beneath it. Evaluating right to left — starting at the loss and walking backwards — computes each shared factor once and reuses it everywhere it appears. Evaluating left to right, or re-deriving the product separately per parameter, recomputes the same sub-expressions over and over. Mechanically the backward pass is strikingly local: each layer receives exactly one thing, the gradient of the loss with respect to its own output, and from that plus the input it saved on the way forward it computes two things — its own weight gradients, and the gradient to hand to the layer below. No layer needs to know anything about the rest of the network.

Direction is the whole result, and it has a name. Automatic differentiation can sweep either way: forward mode costs one pass per input variable, reverse mode one pass per output. Training a network has millions of inputs (the parameters) and exactly one output (the scalar loss), so reverse mode is cheaper by a factor of the parameter count. Reverse the situation — one input, a million outputs — and forward mode would win instead. The general statement is the cheap-gradient principle of Baur and Strassen (1983): the complete gradient of a scalar function costs at most a small constant multiple, around 5x, of evaluating the function once, no matter how many inputs it has. Backpropagation is that theorem applied to a layered network, and "reverse-mode automatic differentiation" is precisely what PyTorch, TensorFlow and JAX implement.

What the algorithm spends instead is memory, and this is the trade almost nobody mentions. Because dz/dw for a layer is that layer's input, the backward pass needs the activations the forward pass produced — so the forward pass must keep them all, and they are freed only as the reverse sweep consumes them. That footprint scales with depth × batch size × layer width, and is unrelated to parameter count. Take 48 layers, a batch of 32 sequences of 1,024 tokens, hidden width 4,096, one saved tensor per layer in bf16: 32 × 1024 × 4096 × 48 × 2 bytes ≈ 12.9 GB. A real transformer block saves several tensors per layer, so the true figure is a multiple of that — and it sits on top of the weights and the optimizer state. Activation memory, not parameters, is usually what makes a batch size stop fitting.

Gradient checkpointing is the direct answer to that bill: keep activations at only a few layer boundaries and recompute the intermediate ones during the backward pass. Saving every √L-th boundary cuts activation memory from O(L) to O(√L) — for 48 layers, roughly 7x less — at the price of about one extra forward pass, moving total training cost from ~3x a forward pass to ~4x ("Training Deep Nets with Sublinear Memory Cost", Chen et al., 2016). Paying 33% more compute for 7x less memory is the standard bargain when a model will not otherwise fit.

Real-World Applications

Every mainstream framework is a backpropagation engine. PyTorch's autograd records each operation of the forward pass into a graph as it executes, then loss.backward() traverses that graph in reverse, applying each operation's known local derivative; TensorFlow's GradientTape and JAX's grad do the same with different ergonomics. The generalization beyond hand-derived layer formulas is what made architecture research cheap: a new layer needs a forward definition and its local derivative, and the framework assembles the rest.

Activation checkpointing is not a research curiosity but standard configuration in large-model training. Megatron-LM and DeepSpeed both expose it, and it is routinely what makes a given model, batch size and device count fit at all — a decision made directly against the memory arithmetic above, and one of the reasons distributed training setups differ so much in achieved throughput.

Exploding gradients are handled in the training loop rather than the architecture: clipping the global gradient norm to a fixed value (1.0 is the common choice, used in GPT-3's published training setup) rescales the whole gradient vector whenever its norm exceeds the threshold, preserving direction while capping step size. Recurrent models get a further variant — truncated backpropagation through time, which unrolls only the last k steps of a sequence rather than all of them, bounding both memory and the depth over which gradients compound in an RNN.

Finally, the finite-difference method that is useless for training is genuinely useful as a test. torch.autograd.gradcheck compares an analytic backward implementation against numerical derivatives and is the standard way to catch a hand-written custom gradient that is subtly wrong — a bug that otherwise shows up only as a model that trains slightly worse than it should.

Key Concepts

• Chain rule: the whole algorithm is dL/dw = dL/da · da/dz · dz/dw applied once per layer; everything else is bookkeeping about what order to evaluate that product in, and what to keep from the last evaluation.
• Reverse mode: differentiating from the single scalar output backwards costs one sweep per output instead of one per input — the reason training a million-parameter model is affordable and differentiating a million outputs would not be.
• Activation memory: the tensors saved on the way in dominate the footprint at large batch sizes, and are released only as the reverse sweep consumes them, which is why peak usage lands at the turnaround point.
• Gradient checking: comparing an analytic derivative against a central finite difference and demanding agreement to roughly 1e-9 is the standard test for a hand-written backward implementation.
• Vanishing and exploding gradients: a product of many per-layer factors either decays toward zero or blows up — a factor of 1.5 sustained across 50 layers gives 1.5⁵⁰ ≈ 6.4 × 10⁸, which is why clipping exists.

Challenges

The failure mode that shaped modern architectures is compounding. Because the backward pass multiplies by one factor per layer, the gradient reaching an early layer is a product of many numbers, and products of many numbers do not stay near 1. Slightly-below-1 factors vanish, slightly-above-1 factors explode, and both get worse with depth. Activation functions carries the arithmetic for the classic sigmoid case; the point here is that this is not a defect of the algorithm but an unavoidable consequence of what the chain rule is. Residual connections, normalization layers and non-saturating activations are all, in part, engineering to keep that product near 1 — which is what made networks of over 100 layers trainable at all in deep learning.

The backward pass is also strongly memory-bound rather than compute-bound. It reads back every stored activation and writes a gradient for every parameter, so on modern accelerators it frequently waits on bandwidth rather than arithmetic — the memory wall shows up in training at least as sharply as in inference, and it is the reason recomputing activations can be faster than fetching them.

Two structural constraints follow from the ordering. First, no backward work can begin until the forward pass has finished, which in pipeline-parallel training leaves devices idle in the well-known pipeline "bubble" and forces micro-batching to fill it. Second, the graph must be differentiable end to end: hard decisions like argmax, sampling and discrete routing have zero or undefined derivatives, so they are handled with surrogates — straight-through estimators, softmax relaxations, or policy-gradient methods that sidestep the chain rule entirely.

Precision is the quiet one. Gradients are typically much smaller in magnitude than activations, so in fp16 they underflow to zero long before the forward values do; mixed-precision training exists partly to solve this, scaling the loss up by a large constant before the backward pass and dividing the gradients back down afterwards. bf16 trades mantissa bits for exponent range specifically to make this less fragile.

Future Trends

The live alternatives are motivated almost entirely by backpropagation's two structural costs — the stored activations and the strict global forward-then-backward ordering. Hinton's forward-forward algorithm (2022) replaces the backward pass with two forward passes and a local objective per layer; feedback alignment and other local learning rules aim at the same target. None is competitive with backpropagation at scale, and it would be a mistake to report them as imminent replacements; they are interesting because they attack a real cost, not because they have paid off.

The biological-plausibility question is the sharpest theoretical criticism and remains open. Backpropagation requires the backward pass to use the same weight values as the forward pass — the weight transport problem — and no known neural circuit has a mechanism for that. "Backpropagation and the brain" (Lillicrap et al., 2020) surveys the approximations that might close the gap.

The more consequential near-term movement is unglamorous: as compute grows faster than memory bandwidth, the recompute-versus-store dial keeps turning toward recompute. Selective and fully automated checkpointing policies, fused backward kernels, and compilers that plan the whole reverse sweep as one tensor operation schedule are where the practical gains are, and they change the constants of the algorithm rather than the algorithm.

Code Example

Backpropagation for a 2-3-1 network, checked against the naive method it replaces. The two must agree — that check is the lesson, and it is also how you debug a real implementation.

import numpy as np

rng = np.random.default_rng(0)
x = rng.normal(size=(1, 2))                    # one example, 2 features
y = np.array([[1.0]])                          # target
W1, b1 = rng.normal(size=(2, 3)) * 0.5, np.zeros((1, 3))
W2, b2 = rng.normal(size=(3, 1)) * 0.5, np.zeros((1, 1))

def forward(W1, b1, W2, b2):
    h = np.tanh(x @ W1 + b1)                   # activation: kept, the backward pass needs it
    out = h @ W2 + b2
    return h, out, float(((out - y) ** 2).mean())

h, out, loss = forward(W1, b1, W2, b2)

# Backward pass: the chain rule applied right to left, ONCE.
d_out = 2.0 * (out - y) / out.size             # dL/d(out)
dW2 = h.T @ d_out                              # (incoming activation) x (outgoing gradient)
d_h = d_out @ W2.T                             # push the gradient back through W2
d_z1 = d_h * (1.0 - h ** 2)                    # times tanh'(z) = 1 - tanh(z)^2
dW1 = x.T @ d_z1                               # same local rule, one layer down

# The naive alternative: nudge one weight, re-run the whole network, repeat per weight.
eps, num, passes = 1e-6, np.zeros_like(W1), 0
for i in range(W1.shape[0]):
    for j in range(W1.shape[1]):
        up, dn = W1.copy(), W1.copy()
        up[i, j] += eps
        dn[i, j] -= eps
        num[i, j] = (forward(up, b1, W2, b2)[2] - forward(dn, b1, W2, b2)[2]) / (2 * eps)
        passes += 2

np.set_printoptions(precision=6, suppress=True)
print("backprop dW1:", dW1.ravel())
print("numeric  dW1:", num.ravel())
print("max difference: %.1e" % abs(dW1 - num).max())
print("forward passes: backprop 1, finite differences %d (for %d weights)" % (passes, W1.size))

Output:

backprop dW1: [ 0.08188   0.146344  0.07187  -0.086031 -0.153764 -0.075514]
numeric  dW1: [ 0.08188   0.146344  0.07187  -0.086031 -0.153764 -0.075514]
max difference: 4.6e-11
forward passes: backprop 1, finite differences 12 (for 6 weights)

The two columns agree to eleven decimal places, so both are computing the same thing. The last line is the reason only one of them is usable: the finite-difference loop scales with the number of weights, and the backward pass does not. Six weights cost twelve extra forward passes here; a billion weights would cost two billion, while the backward pass would still cost one.