Convolution

Definition

A convolution lays a small grid of numbers — the kernel, usually 3×3 — over a patch of the image the same size, multiplies each kernel number by the pixel value underneath it, and adds the nine products into a single number. That one number becomes one pixel of the output; slide the kernel one step right and repeat, across every position, and the sums form a new image called a feature map.

Here is the whole operation on real numbers. Take a vertical-edge kernel, whose arithmetic is "right column minus left column":

-1   0  +1
-1   0  +1
-1   0  +1

Lay it over a flat patch of dark pixels, every one valued 10. The left column contributes −10 three times, the middle column contributes 0, the right column contributes +10 three times, and the total is 0. Now slide one step right, so the patch straddles a boundary between dark and bright pixels:

 10   10  200
 10   10  200
 10   10  200

Same kernel, same procedure: each row gives (−1 × 10) + (0 × 10) + (1 × 200) = 190, and three rows give 570. Flat area → 0. Edge → 570. Nobody told the kernel what an edge is; subtracting the left column from the right returns zero whenever the two match and a large number whenever they differ. Every convolution in every vision model is that operation repeated — one small dot product per output pixel, the same handful of weights reused at every position — and the only thing training changes is which nine numbers are in the grid.

How It Works

Output size: the one formula worth memorising

Running a k×k kernel over an n×n input with padding p and stride s produces an output of

floor((n + 2p - k) / s) + 1

pixels per side. Work it on the 6×6 example above with k = 3, p = 0, s = 1: floor((6 + 0 − 3) / 1) + 1 = 4. A 6×6 image becomes 4×4, because the kernel cannot hang over the border — you lose (k − 1)/2 = 1 pixel from each of the four sides. Stack ten such layers and a 32×32 image erodes to 12×12; the image is being eaten from the outside in.

Padding fixes it. With p = 1, floor((6 + 2 − 3) / 1) + 1 = 6, the size it started at. In general any odd kernel with p = (k − 1)/2 and stride 1 preserves the input size exactly: 3×3 with padding 1, 5×5 with padding 2, 7×7 with padding 3. That is why "3×3, padding 1, stride 1" is the default in essentially every vision network — you can stack fifty of those layers and the feature map is still the resolution you fed in, so depth and resolution become independent design choices.

Stride is deliberate shrinkage. With n = 224, k = 3, p = 1, s = 2: floor((224 + 2 − 3) / 2) + 1 = 111 + 1 = 112, exactly half. It is one of the two ways a network reduces resolution, the other being pooling.

Receptive field: the part people miss

A neuron's receptive field is the region of the original input that can change its value. After one 3×3 layer, it is 3×3. After two, it is 5×5 — the second layer reads nine outputs of the first, each of which already looked at its own 3×3 window, and those windows overlap. After three layers, 7×7. For L stacked 3×3 stride-1 layers the rule is simply:

receptive field = 2L + 1

Depth buys context linearly, one pixel per layer in each direction. Run the numbers on a photograph and the consequence is uncomfortable: ten 3×3 layers give a 21×21 receptive field, which on a 224×224 image is 9% of the width. A neuron ten layers deep cannot see a whole face. Covering all 224 pixels with plain 3×3 layers would take L = 111 layers.

Networks escape this in two ways. The first is downsampling, and the general receptive-field recurrence shows why it works: r₍ₗ₎ = r₍ₗ₋₁₎ + (k − 1) × j₍ₗ₋₁₎, where j is the jump, the product of all strides so far — how many input pixels one step in this layer moves. Each stride-2 layer doubles j, so everything after it grows the receptive field twice as fast. A conv3 → pool2 → conv3 → pool2 → conv3 stack gives receptive fields of 3, 4, 8, 10, 18, where five plain 3×3 layers would give 11.

The second is dilation, which spaces the kernel's taps apart instead of packing them. A 3×3 kernel with dilation d spans k + (k − 1)(d − 1) pixels while still holding only 9 weights: d = 2 spans 5×5, d = 4 spans 9×9, d = 8 spans 17×17. Stack dilations 1, 2, 4, 8 and the receptive field goes 3 → 7 → 15 → 31 in four layers and 36 weights — exponential growth instead of linear. WaveNet used stacked dilated 1D convolutions to reach thousands of audio samples of context, and DeepLab used them to enlarge the receptive field for segmentation without throwing away resolution.

Why two 3×3 layers beat one 5×5

Both see a 5×5 region. But per input/output channel pair, the 5×5 kernel holds 25 weights while two 3×3 kernels hold 2 × 9 = 18, which is 28% fewer. Extend it: three 3×3 layers match a 7×7 receptive field with 27 weights against 49, a 45% saving. And because an activation function sits between the small layers, the stack computes a non-linear function of that 5×5 window where a single large kernel computes a linear one. This is the argument the VGG paper made in 2014, and it is why 3×3 has been the default kernel size ever since.

Channels: where the parameter count really comes from

This is the step where people lose the thread. A colour image has 3 channels, so a kernel for it is not 3×3 = 9 numbers but 3×3×3 = 27 numbers — and it still produces one number per position, summing across all three channels at once. One kernel yields one output channel. Producing 64 output channels therefore requires 64 separate 3×3×3 kernels: 64 × 27 = 1,728 weights, plus one bias each, for 1,792 parameters. PyTorch's nn.Conv2d(3, 64, 3) reports exactly that number.

Deeper in a network the channel counts dominate. A 3×3 layer mapping 256 channels to 256 holds 3 × 3 × 256 × 256 = 589,824 weights. Its compute is H_out × W_out × C_out × (k² × C_in) multiply-accumulates, so at 56×56 resolution that single layer costs 56 × 56 × 256 × 2,304 ≈ 1.85 billion MACs per image. Note the asymmetry: a convolution's parameter count does not depend on image size at all, but its FLOP count is proportional to it.

Depthwise separable convolutions

A standard convolution mixes space and channels in one step. Splitting the two is far cheaper. A depthwise convolution applies one 3×3 kernel per input channel with no channel mixing; a pointwise 1×1 convolution then mixes channels with no spatial extent. Taking the same 56×56, 256 → 256 layer:

• Standard 3×3: 589,824 weights, ~1.85 billion MACs.
• Depthwise 3×3: 9 × 256 = 2,304 weights, 56 × 56 × 256 × 9 = 7.2 million MACs.
• Pointwise 1×1: 256 × 256 = 65,536 weights, 56 × 56 × 256 × 256 = 206 million MACs.
• Separable total: 67,840 weights and 213 million MACs — 8.7× less of both.

The general ratio is 1/C_out + 1/k², here 1/256 + 1/9 = 0.115. For 3×3 kernels the first term is negligible, so the saving is essentially 1/k² ≈ 1/9 whatever the channel count. MobileNet (2017) was built on precisely this substitution, and it is why phones can run vision models at all.

It is not technically convolution

Textbook convolution flips the kernel 180° before the multiply-and-add. What every deep learning framework calls a convolution does not flip anything, which makes the operation cross-correlation. Nobody minds, because the kernel is learned: whichever convention the framework uses, gradient descent stores the nine weights in the orientation that convention needs, and the learned filter is identical up to a flip. It matters in exactly one situation — when you take a hand-designed filter from a signal-processing reference (or scipy.signal.convolve2d, which does flip) and paste it into a network. Asymmetric kernels such as Sobel will then point the wrong way.

Real-World Applications

The sliding dot product predates deep learning and is still run with hand-chosen weights everywhere images are processed. Gaussian blur, unsharp mask and edge detection in image editors and phone camera pipelines are the same operation with fixed kernels: the 3×3 Gaussian [[1,2,1],[2,4,2],[1,2,1]] / 16 blurs because its weights sum to 1 and taper outward. It is also separable in the classical sense — running [1,2,1]/4 horizontally then vertically gives an identical result with 6 multiplies per pixel instead of 9, and the saving grows with kernel size (a 9×9 Gaussian costs 18 multiplies instead of 81, a 4.5× reduction).

Inside GPUs the naive sliding loop is never what runs. cuDNN and similar libraries usually use im2col: copy every k²·C_in patch into a row of a large matrix so the whole layer becomes one matrix multiplication, the operation GPU hardware is fastest at. The copy costs up to k² = 9× the memory of the input feature map, and that trade is accepted routinely. For 3×3 stride-1 layers specifically, the Winograd algorithm F(2×2, 3×3) computes a 2×2 output tile with 16 multiplies where the direct method needs 36, a 2.25× reduction in multiplies, and it is what most 3×3 layers actually execute.

The operation generalises by dimension count rather than by domain. One-dimensional convolutions slide over waveforms and over token embeddings — WaveNet's dilated causal convolutions over raw audio are the same arithmetic on a 1D array — while three-dimensional convolutions slide a k×k×k kernel through CT and MRI volumes and through video as a space-plus-time block. Transposed convolution reverses the geometry to increase resolution, which is how U-Net decoders and image generators get back from a small feature map to a full-size output. On phones and edge devices, the depthwise-separable variant is the reason live segmentation and background blur run inside a power budget of a couple of watts.

Key Concepts

• Kernel (filter): the small weight grid, k×k×C_in numbers. It is the only thing learned; everything else about a convolution is fixed geometry.
• Stride: how far the kernel jumps between positions. s = 2 halves each spatial dimension and quarters the number of output positions, so it cuts a layer's compute by 4×.
• Padding: fabricated border values, almost always zeros, added so the kernel can be centred on edge pixels. p = (k − 1)/2 with stride 1 keeps output size equal to input size.
• Dilation: spacing between kernel taps. Buys receptive field without buying weights — a dilation-8 3×3 kernel spans 17 pixels using 9 numbers.
• Feature map: the grid of numbers one kernel produces across the whole image. A layer with 64 filters emits 64 of them, and they become the next layer's input channels.
• Receptive field: the input region that can influence one output value; 2L + 1 for L stacked 3×3 stride-1 layers, and the quantity to check when a model seems unable to use global context.

Challenges

The memory cost of convolution is dominated by activations, not weights, and that surprises people who reason from parameter counts. A first layer producing 64 channels at 224×224 stores 224 × 224 × 64 ≈ 3.2 million activations per image, which is 12.8 MB in fp32 against 7 KB for the layer's 1,792 weights — a factor of about 1,800. At batch size 32 that one layer's output is 411 MB, and training must keep it alive until the backward pass reaches it. im2col multiplies the same figure by up to k².

Zero padding is a small lie told at every border, and networks notice. Output pixels near the edge are computed partly from invented zeros, and because that pattern is perfectly correlated with position, CNNs learn to read absolute position out of it — the effect was measured directly in How Much Position Information Do Convolutional Neural Networks Encode? (Islam et al., 2020). It is useful when position matters and a hidden source of failure when a model is expected to be position-agnostic.

Strided convolution also breaks the translation invariance the operation is supposed to provide. Subsampling by 2 without a low-pass filter is textbook aliasing, and Making Convolutional Networks Shift-Invariant Again (Zhang, 2019) showed that shifting an input image by a single pixel can flip a classifier's prediction. Transposed convolution has the mirror-image defect: when the kernel size is not divisible by the stride, output pixels receive unequal numbers of contributions and the result carries visible checkerboard artefacts (Odena et al., 2016).

Finally, a FLOP saving is not automatically a speed saving. Depthwise convolutions perform only 9 MACs per element read, against 2,304 for a 256-channel 3×3 layer, so they are memory-bandwidth-bound rather than compute-bound: the 8.7× reduction in arithmetic above typically buys far less than 8.7× in wall-clock time on a GPU built to keep thousands of multipliers fed.

Code Example

A convolution written out in full, with no framework hiding the loop. The image is dark on the left and bright on the right, with one vertical edge down the middle.

import numpy as np

def convolve2d(image, kernel, padding=0, stride=1):
    """Slide `kernel` over `image` and return the map of dot products."""
    k = kernel.shape[0]
    padded = np.pad(image, padding, mode="constant")
    n_out = (padded.shape[0] - k) // stride + 1
    out = np.zeros((n_out, n_out))
    for i in range(n_out):
        for j in range(n_out):
            patch = padded[i * stride:i * stride + k, j * stride:j * stride + k]
            out[i, j] = np.sum(patch * kernel)   # multiply, then add up
    return out

# A 6x6 "image": dark on the left, bright on the right, one vertical edge.
image = np.array([[10, 10, 10, 200, 200, 200]] * 6)

# A vertical-edge detector: right column minus left column.
kernel = np.array([[-1, 0, 1],
                   [-1, 0, 1],
                   [-1, 0, 1]])

print("input:\n", image)
print("\noutput (valid, stride 1):\n", convolve2d(image, kernel).astype(int))
print("\noutput shape with padding=1:", convolve2d(image, kernel, padding=1).shape)

# Two single dot products, done the long way:
for name, patch in [("flat patch", image[0:3, 0:3]), ("edge patch", image[0:3, 1:4])]:
    print("\n" + name + ":\n", patch)
    print("elementwise products:\n", patch * kernel)
    print("sum =", np.sum(patch * kernel))

Output:

input:
 [[ 10  10  10 200 200 200]
 [ 10  10  10 200 200 200]
 [ 10  10  10 200 200 200]
 [ 10  10  10 200 200 200]
 [ 10  10  10 200 200 200]
 [ 10  10  10 200 200 200]]

output (valid, stride 1):
 [[  0 570 570   0]
 [  0 570 570   0]
 [  0 570 570   0]
 [  0 570 570   0]]

output shape with padding=1: (6, 6)

flat patch:
 [[10 10 10]
 [10 10 10]
 [10 10 10]]
elementwise products:
 [[-10   0  10]
 [-10   0  10]
 [-10   0  10]]
sum = 0

edge patch:
 [[ 10  10 200]
 [ 10  10 200]
 [ 10  10 200]]
elementwise products:
 [[-10   0 200]
 [-10   0 200]
 [-10   0 200]]
sum = 570

Three things to read off that output. The 6×6 input became 4×4 without padding and stayed 6×6 with padding 1, matching floor((6 + 2p − 3) / 1) + 1. Flat regions are exactly 0 while the two columns spanning the edge are 570, so the edge has been isolated as a band of large values. And the whole feature map came from nine weights applied 16 times — the same nine numbers everywhere, which is what makes stacking these layers into a convolutional neural network affordable and what makes them the base operation of computer vision.