Residual Connections (Skip Connections)

Definition

A residual connection — also called a skip connection — is a wiring pattern that adds a layer block's input straight back onto its output. Instead of a block computing y = F(x), it computes y = F(x) + x, so the block only has to learn the change F(x) it needs to make to x rather than reproduce the whole signal from scratch. That single addition is what made networks hundreds of layers deep — and every modern transformer — trainable at all. Without it, past a few dozen layers a deeper network gets worse, and not because it overfits.

He et al. found this the hard way. A plain 56-layer convolutional network trained on CIFAR-10 had higher training error than an otherwise identical 20-layer one — not test error, training error, so not overfitting but a plain failure to optimise (Deep Residual Learning, 2015). The same thing happened on ImageNet: a plain 34-layer network reached 28.54% top-1 error against 27.94% for a plain 18-layer one. This is counter-intuitive, because a deeper network can always match a shallower one by setting its extra layers to the identity function and copying the rest — so in principle it should never do worse. Gradient descent simply could not find that solution. Residual connections make it findable by building the identity in for free: a block that needs to do nothing just drives F(x) toward zero, and the input flows through untouched.

How It Works

Write the mapping you actually want a block to compute as H(x). A plain block tries to learn H(x) directly. A residual block instead learns F(x) := H(x) - x — the residual, the part of the answer that is not already sitting in the input — and reconstructs the target by adding the input back: H(x) = F(x) + x. Nothing about the target changed; only what the weights are asked to produce did. This matters because the two starting points are not equally easy. If the best thing a block can do is leave its input alone, a plain block has to coax a stack of nonlinear layers into approximating the identity function, which is hard; a residual block only has to push its weights toward zero, which is where they already start. In He et al.'s words, if an identity mapping were optimal it is easier to push the residual to zero than to fit an identity by a stack of nonlinear layers.

The shortcut is almost free. When the input and output have the same shape, x is added elementwise to F(x) — no weights, no meaningful compute, "neither extra parameter nor computational complexity" in the paper's phrasing. Only where a block deliberately changes the feature dimension (a convolutional stage that halves the spatial size and doubles the channels, say) does the shortcut need a small projection — a 1x1 convolution or linear layer — to make the two tensors addable.

The gradient shortcut

The deeper reason residual connections work lives in the backward pass. Training moves weights by backpropagation, which sends the loss gradient from the output back toward the input through the chain rule — and the chain rule multiplies one local factor per layer. In a plain stack, if each layer's local derivative has magnitude a little under 1 — say 0.8, a plausible value once you are past a saturating nonlinearity — those factors compound. Over 50 layers the gradient reaching the first layer is 0.8^50 ≈ 1.4 × 10⁻⁵ of the gradient at the output: a roughly 70,000x attenuation. The early layers receive a gradient indistinguishable from zero, never move, and the network behaves as if it were shallow. This is the compounding that gradient descent cannot escape, and it is closely related to the classic vanishing-gradient problem that backpropagation covers in detail.

A residual block changes the arithmetic. Because y = x + F(x), the local derivative of the block is 1 + F'(x) — the 1 is the identity branch, and it is not multiplied away. Chain a stack of these together and the product always contains one term in which every factor is the identity: a path worth exactly 1^50 = 1 that carries the downstream gradient back to any earlier layer completely unattenuated. Whatever the residual branches do, the gradient at an early layer is at least the gradient at the output, undiminished by depth. For it to vanish, the residual terms across an entire minibatch would have to conspire to cancel that 1 exactly, which effectively never happens. The ## Code Example below runs both cases and prints the contrast: 1.4 × 10⁻⁵ versus 1.0.

This is why depth stopped being a liability. It is also why the two ideas most often mentioned alongside residual connections — normalization layers, which keep the scale of activations in check, and non-saturating activations like ReLU — are complementary rather than competing: all three exist to keep that per-layer product near 1, and modern architectures use them together.

Real-World Applications

ResNet and the ImageNet leap. The architecture that introduced residual connections, ResNet, won the ILSVRC 2015 classification challenge. Its ensemble reached 3.57% top-5 error, and a single 152-layer model — eight times deeper than the VGG networks that preceded it, yet lower in complexity — reached 4.49% top-5. The paper went on to train a 1,202-layer network on CIFAR-10; it optimised fine (training error under 0.1%) even if it overfit. Before residual connections a 152-layer classifier was not merely hard to train, it was worse than a shallow one. ResNet-50 remains a standard vision backbone and one of the most common benchmarking baselines in the field a decade later.

Every transformer. The reason residual connections are not a computer-vision footnote is that transformers are built from them. Each block wraps its attention sublayer and its feed-forward sublayer in its own residual connection, so the input to the whole stack is added and re-added at every layer. Practitioners call the running sum the residual stream: a shared, fixed-width bus that every block reads from and writes back to. This is a large part of why language models with dozens to well over a hundred layers train without the gradient dying, and the residual stream has become a central object in how transformer internals are studied.

U-Net and medical imaging. The U-Net segmentation architecture uses long skip connections that carry fine spatial detail from its downsampling half directly across to its upsampling half. These concatenate rather than add — a related variant of the same skip idea — and let the decoder recover boundaries the encoder had blurred away. U-Net and its descendants are the standard baseline for segmenting tumours and organs in CT and MRI volumes, and the same encoder-with-skips shape reappears in the denoising networks inside diffusion image generators.

Key Concepts

Degradation is not overfitting, and not exactly vanishing gradients either. The problem residual connections solve is that deeper plain networks get worse on the training set — an optimisation failure, distinct from overfitting, where the network could fit the data and simply fails to. Vanishing gradients are one mechanism behind it, but normalization had already let plain networks of a few dozen layers start converging; degradation is what remained after that, and residual connections are the fix.

Identity versus projection shortcuts. Most shortcuts are pure identity: add the input, change nothing, spend nothing. A projection shortcut appears only where a block changes shape and the raw input cannot be added to the output; it reintroduces a few parameters. He et al. tested using projections everywhere and found it gave a marginal gain for a real cost, which is why the identity shortcut is the default and projections are the exception.

The residual stream has a fixed width. Because every block adds its output back to the same vector, that vector's dimension is constant through the whole stack — 4,096 or 8,192 numbers per token in a large language model, read and written by every one of its layers. This shared bus is powerful but also a constraint: the model's entire working memory at each position is that fixed-width stream.

Challenges

Where to put the normalization. In transformers the placement of the normalization layer relative to the residual addition is a genuine design decision with real consequences. Post-normalization (normalize after adding) tends to give slightly better final quality but is notoriously hard to train deep, often needing careful learning-rate warmup. Pre-normalization (normalize the block's input, add the raw residual) trains far more stably at depth and is now the common choice for large models, but it lets the residual stream's magnitude grow layer over layer, which can cause later layers to contribute less. Neither is free, and the right answer depends on depth and scale.

Residual connections do not remove the need for normalization. They keep the gradient alive, but they do nothing to control the scale of the forward activations, which can still drift. The two techniques address different failure modes, and dropping normalization on the theory that the shortcuts have made training robust reliably destabilises a deep network.

Adding is not always the right merge. Residual addition assumes the input and the block's output live in the same space and should be summed. When that is not true — when you want to preserve two sources of information side by side rather than blend them — concatenation (as in U-Net) can be the better skip, at the cost of a growing feature dimension. Choosing addition by reflex where concatenation was wanted quietly throws information away.

Code Example

A residual block is a one-line change in the forward pass — return x + self.block(x) instead of return self.block(x). The mechanism that makes it matter is easier to see on the backward pass. The script below reduces a deep stack to a single scalar path and compares how much of the output gradient survives back to the first layer, with and without the identity shortcut.

import numpy as np

# One scalar path through a 50-layer stack, to isolate the gradient shortcut.
# Each layer's local derivative has magnitude ~0.8 in the plain net:
# a plausible sub-unit contraction once you are past a saturating nonlinearity.
depth = 50
local = 0.8

# Plain stack: backprop multiplies one factor per layer.
plain = local ** depth
print(f"plain net:    gradient at layer 1 = {plain:.3e} of the output gradient")
print(f"              a {1 / plain:,.0f}x attenuation over {depth} layers")

# Residual stack: each block is y = x + F(x), so the backward factor is (1 + f'),
# and the '1' is the identity branch. The product ALWAYS keeps an unattenuated
# all-identity path worth exactly 1, whatever the residual branch does.
identity_path = 1.0 ** depth
print(f"residual net: identity path alone = {identity_path:.3e} of the output gradient")

Running it as written prints:

plain net:    gradient at layer 1 = 1.427e-05 of the output gradient
              a 70,065x attenuation over 50 layers
residual net: identity path alone = 1.000e+00 of the output gradient

The plain network hands its first layer a gradient 70,000 times smaller than the one at its output — effectively no training signal, which is the degradation He et al. measured. The residual network's identity branch delivers the full gradient regardless of depth. A real block's residual branch adds more paths on top of that guaranteed 1, but the floor it establishes is the whole point: depth can no longer starve the early layers.