Deep Learning

Definition

Deep learning is machine learning in which the model is a chain of simple transformations applied one after another, each one taking the previous one's output as its input — and deep refers to the length of that chain, not the size of the model. The difference that earns the word is arithmetic: adding a layer can multiply how many distinct patterns the network can express, while adding neurons to an existing layer only adds to it.

That is not the intuitive answer, and the intuitive answer is wrong. A shallow network is not weaker in principle: the universal approximation theorems (Cybenko, 1989; Hornik, 1991) prove that a single hidden layer, given enough units, can approximate any continuous function to any accuracy you like. Depth is not about what is representable. It is about how many units the representation costs.

Here is the size of that discount, on a function you can write down. Twenty stacked stages of two ReLU units each — 40 units, 120 numbers in total — represent a sawtooth with 2²⁰ = 1,048,576 straight segments exactly. A network with one hidden layer, using the same activation, cannot draw a curve with more segments than it has units plus one, so it needs at least 1,048,575 units to match. Same function, same activation, a factor of about 26,000 in size — and the factor doubles for every stage you add. This is the formal content of the depth-separation results (Telgarsky, 2016; Montúfar et al., 2014), and the ## Code Example below runs it.

The second half of the definition is what that discount is spent on. Because each layer sees only the layer beneath it, a deep network is forced to build its answer out of intermediate quantities it invented — a hierarchy of learned features rather than one designed by a person. Before 2012, a computer vision system meant an engineer choosing the features: the standard pedestrian detector of the day fed a linear classifier a 3,780-number HOG descriptor per 64×128 window, every one of those numbers the consequence of a human decision about cells, blocks and orientation bins (Dalal & Triggs, 2005). Deep learning did not improve that descriptor. It deleted the job of writing one.

How It Works

Start with what the chain is made of, because one detail decides whether depth means anything at all. Each layer applies a linear map and then a pointwise activation function. Remove the activation and the whole stack collapses: the product W_L ··· W₂W₁ of L weight matrices is itself one matrix of exactly the same shape, so a hundred linear layers compute precisely what one linear layer computes, with a hundred times the parameters and none of the capacity. Depth is not a property of having many matrices. It is a property of putting something non-linear between them, and everything on this page is downstream of that.

With the non-linearity in place, composition compounds. The cleanest demonstration is the tent map t(x) = 2·ReLU(x) − 4·ReLU(x − ½), which folds the interval [0,1] onto itself using two ReLU units and six numbers. Apply it once and you get a curve with 2 straight pieces; feed its output into an identical stage and you get 4; after k stages you have 2ᵏ pieces, because each stage folds every existing piece in half. Meanwhile a single hidden layer of m ReLU units on a scalar input can place at most m breakpoints, so it draws at most m+1 pieces. Ten stages is 20 units against 1,023; twenty stages is 40 units against 1,048,575. One count grows linearly in the parameters, the other exponentially, and that gap is the entire justification for stacking.

The same compounding shows up as a hierarchy of representations when the input is real data. Layer 1 can only see the raw signal, so it can only learn things visible in the raw signal. Layer 2 sees layer 1's outputs, so the cheapest thing for it to learn is combinations of them; layer 3 sees combinations of combinations. Nothing assigns these roles — the network is optimised end to end against a single loss, and the level of structure each depth can reach is simply what that depth can see. What makes this economical is reuse: one learned edge detector is a component of every texture that contains an edge, and one texture detector is a component of every object part that contains it, so the number of expressible concepts grows combinatorially while the parameter count grows layer by layer. Representation learning covers the learned features themselves; the point here is that the hierarchy is what makes learning them affordable.

None of this was news in 2012 — backpropagation was published in 1986 — so it is worth being exact about what changed, because all three things changed at once and no one of them would have sufficed. Data: ILSVRC-2012 offered 1.2 million labelled training images across 1,000 categories, where the benchmarks that preceded it were tens of thousands of images (Caltech-101 held 9,146). More than a hundredfold, and hand-labelled. Hardware: AlexNet ran 90 epochs over those 1.2 million images, about 10⁸ image presentations, at roughly 0.7 billion multiply–accumulates per forward pass and about three times that for a full training step — on the order of 2×10¹⁷ multiply–accumulates for the run. Krizhevsky et al. (2012) report five to six days on two GTX 580 cards, so about 10¹² FLOP/s sustained; at the tens of GFLOP/s a CPU of the day managed on this arithmetic, the same run would have taken the better part of a year. Algorithms: the same paper reports ReLU units reaching 25% training error on CIFAR-10 six times faster than tanh, and uses dropout at p = 0.5 in the fully connected layers to keep 60 million parameters from memorising the training set. Data without the hardware was unusable, hardware without the data would have overfitted, and both without the algorithmic fixes would not have trained.

Real-World Applications

Deep learning displaced what came before it in one specific circumstance: when the features that mattered were too numerous or too subtle for a person to write down, and had enough structure underneath them to be discovered in stages. The applications worth naming are the ones where that happened decisively.

Speech and translation. Until roughly 2012, speech recognition was a pipeline of hand-designed acoustic features feeding statistical models, with specialists maintaining each stage and no shared objective between them. Deep networks replaced the pipeline end to end, so the acoustic front end could be shaped by what the language model actually needed. Google Translate switched from phrase-based statistical translation to a neural system in 2016, and the quality jump was large enough for ordinary users to notice overnight — which essentially never happens in machine translation.

Protein structure prediction. AlphaFold predicts a protein's three-dimensional shape from its amino-acid sequence, a problem that had resisted decades of effort and otherwise costs months of crystallography or cryo-EM per structure. At CASP14 in 2020 it reached a median GDT_TS around 92 out of 100 across all targets, roughly the accuracy of experimental determination, on a benchmark where the previous state of the art sat far below. DeepMind and EMBL-EBI went on to release predicted structures for over 200 million proteins, roughly every sequence in the public databases, so the starting point for a structural question became a database lookup rather than a multi-year experimental programme. See protein folding for the mechanism.

Weather forecasting. GraphCast, published in Science in 2023, produces a 10-day global forecast at 0.25° resolution in under a minute on a single TPU, against roughly an hour of supercomputer time for the conventional physics simulation, and beat that simulation on about 90% of 1,380 verification targets. It is a graph neural network trained on four decades of reanalysis data, with none of the equations of atmospheric motion written into it — the clearest recent instance of a learned model beating a hand-built model of the process, in a domain everyone assumed was safely physics.

Medical imaging. Deep networks read retinal photographs, mammograms and CT scans, and the field moved unusually fast for a regulated one because radiology already held large archives of images paired with recorded outcomes: the labelled data existed before anyone wanted it for training. See AI in healthcare for how those systems are deployed.

Language models. Every current large language model is the same construction at a different scale — a fixed stack of identical transformer blocks applied to a sequence of tokens, trained by the same gradient step. What is worth noticing is where the growth went. Between AlexNet in 2012 and GPT-3 in 2020, parameter count rose roughly 2,900× (60 million to 175 billion) while depth rose about 12× (8 weight layers to 96). Depth bought the initial breakthrough; scale since then has mostly been bought elsewhere.

What unites these is not that the networks are deep for its own sake. In every case the alternative required a person to specify what to look for, no person could, and a hierarchy of learned intermediate features is the mechanism that let the network specify it instead.

Key Concepts

• Depth multiplies, width adds: one more unit in a single hidden layer adds at most one breakpoint to the function; one more layer can double the pieces. 40 units in 20 stages against 1,048,575 in one is the same statement with the numbers filled in.
• Composition needs a non-linearity: the product of L weight matrices is one matrix, so a linear stack of any depth has exactly the capacity of a single layer. The activation function is not a detail of depth, it is the precondition for it.
• Nobody assigns the levels: the hierarchy of edges, textures and parts is not designed and not supervised. It is what each depth can reach given only the layer below and a single loss at the end.
• End-to-end training is the real break with pipelines: a hand-built feature stage is tuned against its own proxy objective, while every layer of a deep network is tuned against the objective you actually care about — which is why early layers change when you change the task.
• Depth is a serial cost: L layers is L dependent matrix multiplications on the critical path. Width shards across devices and shrinks latency; depth shards across devices and does not.

Challenges

The mistake practitioners actually make is reaching for depth where nothing composes. A hierarchy only pays when the target genuinely decomposes — pixels into edges into parts into objects, characters into words into clauses into meaning. Most tabular business data is already features: there is no sub-structure underneath a column called "customer age" for extra layers to discover, so the depth buys capacity you must then regularise away. The arithmetic is unkind. A ten-layer, 512-wide network on 20 input columns carries about 2.1 million parameters; on a 5,000-row dataset that is roughly 420 parameters per training example, and the network's first job becomes not learning the signal but not memorising the table. Gradient-boosted trees (gradient boosting, random forests) routinely beat carefully tuned deep networks on exactly this data — a result reproduced systematically in "Why do tree-based models still outperform deep learning on typical tabular data?" (Grinsztajn et al., 2022). Reaching for depth here costs weeks and loses.

Depth is not free even when the data supports it, and the failure is not the one people expect. He et al. trained a plain 56-layer convolutional network and an otherwise identical 20-layer one on CIFAR-10 and found the deeper network had higher training error — not test error, training error, so not overfitting but an optimisation failure (Deep Residual Learning, 2015). A deeper network can always represent whatever the shallower one represents, by setting the extra layers to the identity; gradient descent simply could not find that solution. Residual connections and normalisation layers exist to make it findable, and without them extra depth actively hurts. The underlying reason is the compounding of per-layer factors in the backward pass — a product that decays or explodes rather than staying near 1, which backpropagation works through.

Depth costs latency in a way width does not. A forward pass through 80 layers is 80 dependent rounds of weight reads and kernel launches, and no amount of hardware removes the dependency: distributed training and pipeline parallelism raise throughput by keeping more devices busy, but a single request still waits for all 80 rounds. This is why inference on a deep model is so often bandwidth-bound rather than compute-bound (see the memory wall), and why architects reach for width, more experts or a larger vocabulary before they reach for more layers.

And the intermediate features have no names. A hand-built pipeline can be inspected at every stage because a person defined what each stage measures. A learned hierarchy cannot: layer 40's output is a vector, and there is no guarantee any coordinate of it corresponds to a concept a human would recognise. That is the structural source of the interpretability problem — see explainable AI — and it is a direct consequence of the thing that made the approach work.

Future Trends

Depth stopped growing, and almost nobody remarks on it. Image classifiers went from 8 weight layers in 2012 to 19 in VGG to 152 in ResNet by 2015, and then the curve flattened: GPT-3 was 96 layers in 2020, and Llama 3's 70B model 80 layers in 2024, both shallower than a 2015 image model. Ten years of scaling went into width, vocabulary, data and expert count instead. Scaling laws are part of the reason — they prescribe a parameter count N and a token count D, and say almost nothing about how to split N between depth and width, leaving the split to be decided by latency, which favours width.

The growth in depth moved to inference time. A 64-layer model that emits 1,000 reasoning tokens has applied 64,000 layer evaluations in sequence over a shared context — an unrolled recurrence deeper than any stack anyone trains. That is what test-time compute and chain-of-thought buy: extra depth purchased per query at run time rather than once at training time, with the advantage that you can spend more of it on a hard question and none on an easy one.

And the cheapest remaining axis is width without cost. Mixture-of-experts adds parameters that most tokens never touch, raising capacity without raising either depth or per-token FLOPs — the architectural expression of the same trade this page opened with. Depth is what made the field work and it remains expensive to buy more of; width is what the field buys now.

Code Example

The claim in the Definition is checkable in fifteen lines. Each pass through tent is one more layer of two ReLU units; the piece count is measured, not asserted, by counting where the slope changes.

import numpy as np

relu = lambda v: np.maximum(v, 0.0)

# One stage of depth: the tent map, t(x) = 2*relu(x) - 4*relu(x - 0.5).
# Two ReLU units. Six numbers: two weights in, two biases, two weights out.
tent = lambda v: 2.0 * relu(v) - 4.0 * relu(v - 0.5)

def linear_pieces(y):
    """Straight segments of a sampled piecewise-linear curve = slope changes + 1."""
    return 1 + int(np.count_nonzero(np.abs(np.diff(np.diff(y))) > 1e-9))

x = np.linspace(0.0, 1.0, 2**16 + 1)   # grid aligned to powers of two
y = x.copy()

print(f"{'stages':>6}{'units':>7}{'weights':>9}{'pieces':>9}{'units if 1 layer':>18}")
for k in range(1, 11):
    y = tent(y)                        # add one layer
    print(f"{k:>6}{2*k:>7}{6*k:>9}{linear_pieces(y):>9,}{2**k - 1:>18,}")

Output:

stages  units  weights   pieces  units if 1 layer
     1      2        6        2                 1
     2      4       12        4                 3
     3      6       18        8                 7
     4      8       24       16                15
     5     10       30       32                31
     6     12       36       64                63
     7     14       42      128               127
     8     16       48      256               255
     9     18       54      512               511
    10     20       60    1,024             1,023

Read the last two columns against each other. The left one is what 20 units and 60 weights actually produce; the right one is the number of units a single hidden layer would need to produce the same thing. Both columns are exact, and one of them doubles every row while the cost of producing it goes up by two units. Extend the loop to 20 and the deep network is still 40 units while the shallow one passes a million — which is what "depth" is worth, stated as a number rather than an intuition.