---
source: 'https://howaiworks.ai/glossary/residual-connections'
section: glossary
title: Residual Connections (Skip Connections)
description: >-
  Residual connections add a block's input back to its output — F(x) + x — the
  trick that lets gradients flow through deep nets and made transformers
  trainable.
tags:
  - Deep Learning
  - Neural Networks
  - Computer Vision
  - transformer
  - CNN
  - AI Architecture
category: Deep Learning
datePublished: '2026-07-25'
lastUpdated: '2026-07-25'
---

# Residual Connections (Skip Connections)

> Residual connections add a block's input back to its output — F(x) + x — the trick that lets gradients flow through deep nets and made transformers trainable.

## Definition

A residual connection — also called a skip connection — is a wiring pattern that adds a
layer block's input straight back onto its output. Instead of a block computing `y = F(x)`,
it computes `y = F(x) + x`, so the block only has to learn the *change* `F(x)` it needs to
make to `x` rather than reproduce the whole signal from scratch. That single addition is what
made networks hundreds of layers deep — and every modern transformer — trainable at all.
Without it, past a few dozen layers a deeper network gets *worse*, and not because it overfits.

He et al. found this the hard way. A plain 56-layer convolutional network trained on CIFAR-10
had *higher* training error than an otherwise identical 20-layer one — not test error, training
error, so not [overfitting](https://howaiworks.ai/glossary/overfitting) but a plain failure to optimise ([Deep
Residual Learning, 2015](https://arxiv.org/abs/1512.03385)). The same thing happened on
ImageNet: a plain 34-layer network reached 28.54% top-1 error against 27.94% for a plain
18-layer one. This is counter-intuitive, because a deeper network can always match a shallower
one by setting its extra layers to the identity function and copying the rest — so in principle
it should never do worse. Gradient descent simply could not find that solution. Residual
connections make it findable by building the identity in for free: a block that needs to do
nothing just drives `F(x)` toward zero, and the input flows through untouched.

## How It Works

Write the mapping you actually want a block to compute as `H(x)`. A plain block tries to learn
`H(x)` directly. A residual block instead learns `F(x) := H(x) - x` — the *residual*, the part
of the answer that is not already sitting in the input — and reconstructs the target by adding
the input back: `H(x) = F(x) + x`. Nothing about the target changed; only what the weights are
asked to produce did. This matters because the two starting points are not equally easy. If the
best thing a block can do is leave its input alone, a plain block has to coax a stack of
nonlinear layers into approximating the identity function, which is hard; a residual block only
has to push its weights toward zero, which is where they already start. In He et al.'s words, if
an identity mapping were optimal it is easier to push the residual to zero than to fit an
identity by a stack of nonlinear layers.

The shortcut is almost free. When the input and output have the same shape, `x` is added
elementwise to `F(x)` — no weights, no meaningful compute, "neither extra parameter nor
computational complexity" in the paper's phrasing. Only where a block deliberately changes the
feature dimension (a convolutional stage that halves the spatial size and doubles the channels,
say) does the shortcut need a small projection — a 1x1 convolution or linear layer — to make the
two tensors addable.

### The gradient shortcut

The deeper reason residual connections work lives in the backward pass. Training moves weights
by [backpropagation](https://howaiworks.ai/glossary/backpropagation), which sends the loss gradient from the output
back toward the input through the chain rule — and the chain rule *multiplies* one local factor
per layer. In a plain stack, if each layer's local derivative has magnitude a little under 1 —
say 0.8, a plausible value once you are past a saturating nonlinearity — those factors compound.
Over 50 layers the gradient reaching the first layer is `0.8^50 ≈ 1.4 × 10⁻⁵` of the gradient at
the output: a roughly 70,000x attenuation. The early layers receive a gradient indistinguishable
from zero, never move, and the network behaves as if it were shallow. This is the compounding
that [gradient descent](https://howaiworks.ai/glossary/gradient-descent) cannot escape, and it is closely related to
the classic vanishing-gradient problem that [backpropagation](https://howaiworks.ai/glossary/backpropagation) covers
in detail.

A residual block changes the arithmetic. Because `y = x + F(x)`, the local derivative of the
block is `1 + F'(x)` — the `1` is the identity branch, and it is not multiplied away. Chain a
stack of these together and the product always contains one term in which every factor is the
identity: a path worth exactly `1^50 = 1` that carries the downstream gradient back to any
earlier layer completely unattenuated. Whatever the residual branches do, the gradient at an
early layer is at least the gradient at the output, undiminished by depth. For it to vanish, the
residual terms across an entire minibatch would have to conspire to cancel that `1` exactly,
which effectively never happens. The `## Code Example` below runs both cases and prints the
contrast: `1.4 × 10⁻⁵` versus `1.0`.

This is why depth stopped being a liability. It is also why the two ideas most often mentioned
alongside residual connections — normalization layers, which keep the scale of activations in
check, and non-saturating activations like ReLU — are complementary rather than competing: all
three exist to keep that per-layer product near 1, and modern architectures use them together.

## Real-World Applications

**ResNet and the ImageNet leap.** The architecture that introduced residual connections, ResNet,
won the ILSVRC 2015 classification challenge. Its ensemble reached 3.57% top-5 error, and a
single 152-layer model — eight times deeper than the VGG networks that preceded it, yet lower in
complexity — reached 4.49% top-5. The paper went on to train a 1,202-layer network on CIFAR-10;
it optimised fine (training error under 0.1%) even if it overfit. Before residual connections a
152-layer classifier was not merely hard to train, it was worse than a shallow one. ResNet-50
remains a standard vision backbone and one of the most common benchmarking baselines in the
field a decade later.

**Every transformer.** The reason residual connections are not a computer-vision footnote is
that [transformers](https://howaiworks.ai/glossary/transformer) are built from them. Each block wraps its attention
sublayer and its feed-forward sublayer in its own residual connection, so the input to the whole
stack is added and re-added at every layer. Practitioners call the running sum the *residual
stream*: a shared, fixed-width bus that every block reads from and writes back to. This is a
large part of why language models with dozens to well over a hundred layers train without the
gradient dying, and the residual stream has become a central object in how transformer internals
are studied.

**U-Net and medical imaging.** The U-Net segmentation architecture uses long skip connections
that carry fine spatial detail from its downsampling half directly across to its upsampling half.
These *concatenate* rather than add — a related variant of the same skip idea — and let the
decoder recover boundaries the encoder had blurred away. U-Net and its descendants are the
standard baseline for segmenting tumours and organs in CT and MRI volumes, and the same
encoder-with-skips shape reappears in the denoising networks inside diffusion image generators.

## Key Concepts

**Degradation is not overfitting, and not exactly vanishing gradients either.** The problem
residual connections solve is that deeper plain networks get worse on the *training* set — an
optimisation failure, distinct from overfitting, where the network could fit the data and simply
fails to. Vanishing gradients are one mechanism behind it, but normalization had already let
plain networks of a few dozen layers start converging; degradation is what remained after that,
and residual connections are the fix.

**Identity versus projection shortcuts.** Most shortcuts are pure identity: add the input,
change nothing, spend nothing. A projection shortcut appears only where a block changes shape and
the raw input cannot be added to the output; it reintroduces a few parameters. He et al. tested
using projections everywhere and found it gave a marginal gain for a real cost, which is why the
identity shortcut is the default and projections are the exception.

**The residual stream has a fixed width.** Because every block adds its output back to the same
vector, that vector's dimension is constant through the whole stack — 4,096 or 8,192 numbers per
token in a large language model, read and written by every one of its layers. This shared bus is
powerful but also a constraint: the model's entire working memory at each position is that
fixed-width stream.

## Challenges

**Where to put the normalization.** In transformers the placement of the normalization layer
relative to the residual addition is a genuine design decision with real consequences.
Post-normalization (normalize after adding) tends to give slightly better final quality but is
notoriously hard to train deep, often needing careful learning-rate warmup. Pre-normalization
(normalize the block's input, add the raw residual) trains far more stably at depth and is now
the common choice for large models, but it lets the residual stream's magnitude grow layer over
layer, which can cause later layers to contribute less. Neither is free, and the right answer
depends on depth and scale.

**Residual connections do not remove the need for normalization.** They keep the gradient alive,
but they do nothing to control the *scale* of the forward activations, which can still drift.
The two techniques address different failure modes, and dropping normalization on the theory
that the shortcuts have made training robust reliably destabilises a deep network.

**Adding is not always the right merge.** Residual addition assumes the input and the block's
output live in the same space and should be summed. When that is not true — when you want to
preserve two sources of information side by side rather than blend them — concatenation (as in
U-Net) can be the better skip, at the cost of a growing feature dimension. Choosing addition by
reflex where concatenation was wanted quietly throws information away.

## Code Example

A residual block is a one-line change in the forward pass — `return x + self.block(x)` instead
of `return self.block(x)`. The mechanism that makes it matter is easier to see on the backward
pass. The script below reduces a deep stack to a single scalar path and compares how much of the
output gradient survives back to the first layer, with and without the identity shortcut.

```python
import numpy as np

# One scalar path through a 50-layer stack, to isolate the gradient shortcut.
# Each layer's local derivative has magnitude ~0.8 in the plain net:
# a plausible sub-unit contraction once you are past a saturating nonlinearity.
depth = 50
local = 0.8

# Plain stack: backprop multiplies one factor per layer.
plain = local ** depth
print(f"plain net:    gradient at layer 1 = {plain:.3e} of the output gradient")
print(f"              a {1 / plain:,.0f}x attenuation over {depth} layers")

# Residual stack: each block is y = x + F(x), so the backward factor is (1 + f'),
# and the '1' is the identity branch. The product ALWAYS keeps an unattenuated
# all-identity path worth exactly 1, whatever the residual branch does.
identity_path = 1.0 ** depth
print(f"residual net: identity path alone = {identity_path:.3e} of the output gradient")
```

Running it as written prints:

```
plain net:    gradient at layer 1 = 1.427e-05 of the output gradient
              a 70,065x attenuation over 50 layers
residual net: identity path alone = 1.000e+00 of the output gradient
```

The plain network hands its first layer a gradient 70,000 times smaller than the one at its
output — effectively no training signal, which is the degradation He et al. measured. The
residual network's identity branch delivers the full gradient regardless of depth. A real block's
residual branch adds more paths on top of that guaranteed `1`, but the floor it establishes is
the whole point: depth can no longer starve the early layers.

## Frequently Asked Questions

### What is a residual connection?

A wiring pattern that adds a layer block's input straight back onto its output, so the block computes F(x) + x instead of F(x). The block then only has to learn the change F(x) to make to x, and if no change is needed it can drive F(x) to zero and pass the input through unaltered.

### What is the difference between a residual connection and a skip connection?

They are the same idea and the terms are used interchangeably. 'Residual connection' emphasises that the block learns a residual F(x) = H(x) - x; 'skip connection' emphasises that the signal skips past the block. U-Net uses long skip connections that concatenate rather than add, which is a related but distinct variant.

### Why do residual connections help train deep networks?

They give the backward pass an unattenuated shortcut. In a plain stack the gradient is multiplied by one factor per layer and shrinks toward zero with depth; the identity branch of a residual block passes the gradient through with a factor of 1, so every layer receives a usable gradient however deep the network is.

### Do residual connections add parameters or compute?

An identity shortcut adds neither — it is a single elementwise addition. A projection shortcut, needed only where the block changes the feature dimension, adds a small 1x1 convolution or linear layer, but the vast majority of shortcuts in a network are free.

### Do transformers use residual connections?

Yes. Every transformer block wraps both its attention sublayer and its feed-forward sublayer in a residual connection. This is a large part of why models with dozens to over a hundred layers can be trained at all, and why the 'residual stream' is central to how transformers are analysed.

## Related

### Related terms

- [Deep Learning](https://howaiworks.ai/glossary/deep-learning)
- [Backpropagation](https://howaiworks.ai/glossary/backpropagation)
- [Convolutional Neural Network (CNN)](https://howaiworks.ai/glossary/convolutional-neural-network)
- [Transformer](https://howaiworks.ai/glossary/transformer)
- [Neural Network](https://howaiworks.ai/glossary/neural-network)

---

Source: https://howaiworks.ai/glossary/residual-connections — HowAIWorks.ai