Neural Network

Definition

A neural network is a stack of layers, and each layer does exactly two things: it multiplies the list of numbers coming in by a table of numbers, then bends each result with one fixed, simple curve — for example, replacing every negative value with zero. Repeat that pair of steps a few times and you have a neural network. There is nothing else inside it. No rules, no if-statements, no stored examples to look up. The only difference between a network that recognises faces and one that writes text is the shape of the tables and the values in them, and those values are found by training on examples rather than written by a person.

The bending step is the part that is easy to skip over and impossible to do without. Two multiplications in a row are one multiplication. If the first layer's table is [[1, 2], [3, 4]] and the second's is [[5, 6], [7, 8]], then the single table [[19, 22], [43, 50]] does the work of both — for every possible input, exactly, forever. A hundred layers with nothing between them therefore have precisely the expressive power of one, and cost a hundred times as much. The whole case for depth rests on the bend, which is what an activation function is.

Put the bend back and something disproportionate happens. Two hidden units and nine numbers are enough to compute a function that a network with no hidden layer cannot compute at all — and that gap, compounded across dozens of layers and billions of numbers, is the entire distance between a spreadsheet formula and a system that transcribes speech. This page works both halves out on small numbers you can check.

How It Works

One layer, in one line

A layer holds a rectangular table of weights and a short list of biases. It computes bend(x @ W + b), where x is the vector of numbers arriving, @ is matrix multiplication, and bend is applied to each output separately. A single output of that operation is a neuron; the count of individual numbers in W and b is the parameter count. Those two pages cover the unit and the layer in detail. This one is about what happens when you put several in a row and train the result as a single object.

The collapse: why depth without a nonlinearity buys nothing

Take two layers with no bend and no biases. Layer one is W1 = [[1, 2], [3, 4]] and layer two is W2 = [[5, 6], [7, 8]]. Feed in x = [1, 1].

The first layer gives [1×1 + 1×3, 1×2 + 1×4] = [4, 6]. The second turns that into [4×5 + 6×7, 4×6 + 6×8] = [62, 72].

Now multiply the two tables together first. W1 @ W2 is [[1×5 + 2×7, 1×6 + 2×8], [3×5 + 4×7, 3×6 + 4×8]] = [[19, 22], [43, 50]]. Feed the same x = [1, 1] into that single table: [19 + 43, 22 + 50] = [62, 72]. The same answer, from one table instead of two — and not by luck for this input. Matrix multiplication is associative, so (x @ W1) @ W2 = x @ (W1 @ W2) for every x there is. Biases fold in the same way: (x @ W1 + b1) @ W2 + b2 equals x @ (W1 @ W2) + (b1 @ W2 + b2), one table and one bias vector again.

That is the whole result, and it is worth sitting with, because most explanations of neural networks skip it. Depth, on its own, is free and worthless. Stacking linear layers adds parameters and arithmetic without adding a single function the network can compute.

Now insert ReLU — replace negatives with zero — between the two layers, and watch the same numbers stop agreeing. Feed in x = [1, -1]. The first layer gives [1 - 3, 2 - 4] = [-2, -2]; ReLU turns that into [0, 0]; the second layer outputs [0, 0]. The collapsed single table gives [19 - 43, 22 - 50] = [-24, -28] instead.

One disagreement could be patched by choosing a different table, so here is why no table works. A linear map must satisfy f(a + b) = f(a) + f(b). With ReLU in place, f([1, 1]) = [62, 72] and f([1, -1]) = [0, 0], so their sum is [62, 72]. But [1, 1] + [1, -1] = [2, 0], and running that through gives [2, 4] after ReLU and [38, 44] out — not [62, 72]. No matrix, of any size, reproduces this two-layer network. The nonlinearity is not a detail of the implementation. It is the thing that makes a stack more than its product.

XOR: the smallest problem depth solves

XOR outputs 1 when two binary inputs differ and 0 when they agree. A network with no hidden layer computes step(w₁x₁ + w₂x₂ + b), and it cannot do this. Suppose it could. From (0,0) → 0 we get b ≤ 0. From (1,0) → 1 we get w₁ + b > 0, and from (0,1) → 1, w₂ + b > 0. Add those two: w₁ + w₂ + 2b > 0, so w₁ + w₂ + b > -b ≥ 0. But (1,1) → 0 demands w₁ + w₂ + b ≤ 0. The four requirements contradict each other, so no weights exist. Geometrically: the two 1s sit on opposite corners of a square and no straight line separates them from the two 0s. This is the same wall that stops logistic regression, and it is not a wall you can climb by adding more units to that one layer.

One hidden layer of two ReLU units clears it with nine numbers. Take a first layer with weights [[1, 1], [1, 1]] and biases [0, -1], and an output layer with weights [1, -2] and bias 0. Both hidden units see the same quantity s = x₁ + x₂; the second one has its bias shifted so that it only wakes up once s exceeds 1. Trace all four cases:

The last row is the trick. On the way from s = 1 to s = 2 the output would have kept climbing, except that the second hidden unit switched on at s = 1 and started subtracting twice as fast as the first unit adds. The network draws a tent: up, then down. A single layer can only draw a ramp, and XOR needs a tent. Every extra hidden unit adds another possible kink, and every extra layer lets kinks be built out of kinks — which is where the exponential returns on depth described under deep learning come from.

What universal approximation does and does not promise

There is a theorem people reach for at this point, and it is usually overstated. Hornik, Stinchcombe and White (Neural Networks, 2, 359–366, 1989) proved that "standard multilayer feedforward networks with as few as one hidden layer using arbitrary squashing functions are capable of approximating any Borel measurable function from one finite dimensional space to another to any desired degree of accuracy, provided sufficiently many hidden units are available."

Read the clause in bold, because everything hard about neural networks lives in it. The theorem is an existence result. It does not say how many units "sufficiently many" is — the authors state plainly that their results do not address that question — and it says nothing whatsoever about whether gradient descent will find those units' weights from a random start. Both gaps are real. The ## Code Example below trains the two-unit XOR network from thirty random starting points and two of them never get there, on a problem with four training examples and nine parameters. Representability is not attainability.

Training the whole stack as one object

A network is not assembled from separately trained parts. It is trained end to end: one loss function at the output, and every parameter in every layer adjusted against that single number.

The loop has four steps. Run the input forward through every layer and get a prediction. Compare it with the target to get a loss. Work backwards through the layers, computing for each parameter how much the loss would change if that parameter changed slightly. Nudge every parameter a small step in the direction that reduces the loss, then repeat. Machine learning covers the fitting-from-examples idea in general; what is specific to a network is the third step, and it has a name — credit assignment. A weight in layer 1 has no direct contact with the loss. Its influence is filtered through everything above it, and backpropagation is the chain rule applied along that path.

The reason it is done this way is cost, and the arithmetic is stark. You could estimate each gradient by brute force: nudge one parameter, run the whole network forward again, see how the loss moved. For the nine-parameter XOR network that is 10 forward passes per training step. For a network with a million parameters it is 1,000,001. Backpropagation gets the gradient for every parameter from one forward pass plus one backward sweep, whose cost is a small multiple of the forward pass and does not grow with the parameter count. That single property is why networks with hundreds of billions of parameters can be trained at all, and it is a stronger claim about backpropagation than "it updates the weights".

Types

The four names below are a real taxonomy — practitioners and papers use these words — and what separates them is not difficulty or vintage. It is the assumption each one builds into its wiring about the shape of the data. An assumption that is true saves an enormous number of parameters and training examples; an assumption that is false costs accuracy that no amount of data recovers.

Feedforward (fully connected). Information moves in one direction, and every input connects to every unit. The assumption is that there is no assumption: no coordinate of the input is known in advance to be related to any other. A fully connected network would learn just as well from an image whose pixels had been shuffled, as long as the same shuffle were applied to every image. That generality is why it is the default for tabular data — and why it is wasteful on anything with structure.

Convolutional. A convolutional network applies the same small filter at every position of a grid. The assumption is translation equivariance plus locality: a pattern means the same thing wherever it appears, and neighbouring positions are related while distant ones mostly are not. That is true of photographs and audio spectrograms and false of a spreadsheet, where column 3 being next to column 4 means nothing.

Recurrent. A recurrent network applies one weight matrix repeatedly along a sequence, carrying a hidden state from step to step. The assumption is stationarity in time: the rule that turns "state so far plus next item" into "new state" is the same at step 2 and at step 2,000. That lets one small matrix handle a sequence of any length, at the price of putting one multiplication between every step and the loss.

Transformer. A transformer treats its input as a set of elements and computes, from the content of the elements themselves, which ones should influence which. The assumption is that the relevant relationships are data-dependent rather than fixed by position — the opposite of the convolutional bet. Because nothing in the mechanism knows where an element sat, position has to be handed back to the model explicitly. See self-attention for how the weighting is computed, and large language models for what the architecture became.

Real-World Applications

Cheque reading, deployed 1996. LeCun, Bottou, Bengio and Haffner's Gradient-Based Learning Applied to Document Recognition contains the cleanest controlled comparison of a stack against a single layer anywhere in the literature, because both were fitted to the same 60,000 handwritten digits. A linear classifier — one layer, 7,850 parameters — gets 12% of the test set wrong. Hand-engineering the input by deslanting the images brings it to 8.4%. LeNet-5, a stack of convolutional and fully connected layers with roughly 60,000 trainable parameters, reaches 0.95%: about a ninth of the errors of the hand-helped linear model and a thirteenth of the plain one, on identical data. The paper is not a benchmark exercise. The system was integrated into NCR's cheque readers, fielded in US banks from June 1996, and by the paper's own account was "reading millions of checks per day".

Speech recognition, 2012. Hinton and eleven co-authors from four labs reported in Deep Neural Networks for Acoustic Modeling in Speech Recognition that on the Switchboard benchmark a deep network cut word error rate from the 27.4% of the incumbent Gaussian-mixture system to 18.5%, a 33% relative reduction, trained on the same 309 hours of audio. The row of their table that matters most for this page is the control. One hidden layer of 4,634 units reading a window of neighbouring frames — 45.1 million parameters — scores 25.7%. Seven hidden layers of 2,048 units, also 45.1 million parameters, score 19.6%. Same parameter budget, same data, same input window — six points of absolute error separating a shallow arrangement of those parameters from a deep one. This is the empirical answer to the universal approximation theorem's silence: representable and reachable are different.

Machine translation, 2016. Google's GNMT replaced a phrase-based translation pipeline — separate alignment, phrase table, reordering and language models, each tuned by different people against different objectives — with one network of 8 encoder and 8 decoder layers trained end to end against a single loss. Human side-by-side evaluation on production traffic put the reduction in translation errors at around 60% against the system it replaced, across English↔French, English↔Spanish and English↔Chinese. The lesson is less about depth than about the training regime this page describes: when the whole pipeline is one differentiable object, every stage is optimised for what the final output needs rather than for a proxy someone invented for it.

These are not curiosities from three separate fields. All three are the same construction — matrices, a bend, a single loss, one backward sweep — pointed at different inputs.

Key Concepts

Composition, not complication. The individual operation a network performs is simpler than most spreadsheet formulas. What makes it capable is that the output of one such operation becomes the input to the next, so layer 3 works with quantities that layer 2 invented and that nobody named. This is why "the network learned to detect edges" is a description of layer 1 and not of the network: no single layer holds the answer.

End to end means one loss. Every parameter, in every layer, is adjusted against the same scalar. There is no separate objective for the early layers telling them to find good features — the features are whatever happens to reduce the final error. That is simultaneously the source of the method's power and the reason it needs so much data: nobody has told the early layers what to look for.

Everything in the path must be differentiable. Because credit flows backwards through the chain rule, every operation between a parameter and the loss must have a usable derivative. This constraint quietly shapes the whole field. It is why ReLU replaced step functions, why "pick the highest-scoring word" is not a training-time operation, and why anything genuinely discrete — a database lookup, a tool call, a sampled token — has to be handled by a different mechanism such as reinforcement learning.

Challenges

A network can be perfectly capable of a function it will never learn. The universal approximation result guarantees a setting of the weights exists; nothing guarantees gradient descent from a random start reaches it. The ## Code Example shows this at absurdly small scale: two hidden units, four training examples, nine parameters, and 2 of 30 random initialisations settle at a loss of about 0.125 and stay there. Practitioners meet the same phenomenon as "it trains fine with seed 0 and not with seed 1". When a model underperforms, "the architecture cannot represent this" and "the optimiser did not find it" call for opposite responses, and only the training loss tells them apart.

Extrapolation is the failure that looks most like an answer. A ReLU network is exactly a piecewise-straight function: within any region where no unit changes state, it is a single linear map. Past the outermost boundary there are no more kinks to come, so the function simply continues in a straight line forever. The XOR network above makes this concrete. For any inputs with s = x₁ + x₂ ≥ 1 its output is s − 2(s − 1) = 2 − s. Give it x = (2.5, 2.5) and it confidently returns −3; at (3, 3) it returns −4; it will happily return −1,000. Nothing in the arithmetic knows the network was only ever shown the corners of a unit square, and nothing in the output flags the answer as an extrapolation. This is the mechanism behind a demand forecaster that produces negative sales for an unprecedented week.

More units is not monotonically better. Two separate things go wrong. Statistically, capacity you cannot constrain gets spent memorising: the same parameters that let a network fit the signal let it fit the noise, which is overfitting and why regularization exists. Structurally, extra width is often the wrong purchase — the Switchboard comparison above put 45.1 million parameters into one wide hidden layer and into seven narrower ones, and the wide arrangement was six points worse. "Add units until it works" is a strategy that costs money on the way to the wrong answer.

The learned weights are not an explanation, and cannot be made into one by staring harder. Two of the seeds in the code example below both solve XOR exactly, with first-layer weights [0.81, 1.71, 0.81, 1.74] and [1.58, -0.78, 3.18, -0.95]. Same function, unrelated numbers. This is structural rather than accidental: swapping two hidden units along with their outgoing weights leaves the computed function identical, so a layer of 2,048 units has 2,048! — a number with 5,895 digits — exactly equivalent weight settings, before counting rescalings. Any story told about the value of one weight is a story about an arbitrary member of that set. Serious interpretability work therefore looks for features and circuits spanning many units rather than reading individual parameters; see explainable AI and this write-up of sparse-circuit models.

Future Trends

The structural assumptions are being learned rather than built in. Convolution encodes translation equivariance in the wiring; a transformer encodes almost nothing and works out from data which elements relate to which. Vision transformers cut an image into patches and let attention discover locality, matching and then beating convolutional networks once the training set is large enough. The direction of travel is that hand-built inductive bias is a subsidy you pay when data is scarce and a constraint you pay for when it is not — which makes "how much structure to build in" an economic question about your dataset rather than an architectural conviction.

What sits outside the gradient is where the interesting work now is. End-to-end differentiability is the property that made this whole method work, and it excludes exactly the operations that modern systems need most: choosing a tool, retrieving a document, taking a discrete action, deciding to think longer. Reinforcement learning, learned reward models and search over sampled outputs are all ways of training around a step that has no derivative. Expect the boundary of "the differentiable part" to keep being redrawn.

Interpretability is turning from weights to circuits. The permutation argument above means individual weights are not stable objects to study, and the field has responded by looking for structures that are: directions in activation space, sparse features recovered by autoencoders, and small subnetworks that implement identifiable behaviours. This matters commercially and not only philosophically, because "why did it output that" is a question auditors and regulators ask about deployed networks and the honest current answer is a research programme.

Code Example

Nine parameters, two layers, and four training examples are enough to demonstrate both halves of this page: that the nonlinearity is what makes depth mean anything, and that a network which can represent a function may still fail to learn it.

import numpy as np

X = np.array([[0., 0.], [0., 1.], [1., 0.], [1., 1.]])
y = np.array([[0.], [1.], [1.], [0.]])          # XOR: 1 when the two inputs differ

def train(seed, nonlinear, steps=8000, lr=0.2):
    """Two layers and 9 parameters, trained end to end by gradient descent."""
    rng = np.random.default_rng(seed)
    W1, b1 = rng.normal(size=(2, 2)) * 0.5, np.zeros((1, 2))
    W2, b2 = rng.normal(size=(2, 1)) * 0.5, np.zeros((1, 1))
    act = np.tanh if nonlinear else (lambda z: z)
    for _ in range(steps):
        h = act(X @ W1 + b1)                    # forward, layer 1
        out = h @ W2 + b2                       # forward, layer 2
        d_out = 2 * (out - y) / len(X)          # how the loss moves with the output
        dh = (d_out @ W2.T) * ((1 - h ** 2) if nonlinear else 1.0)   # chain rule, one layer back
        gW2, gb2 = h.T @ d_out, d_out.sum(0, keepdims=True)
        gW1, gb1 = X.T @ dh, dh.sum(0, keepdims=True)
        W2, b2 = W2 - lr * gW2, b2 - lr * gb2   # every parameter moves on the same signal
        W1, b1 = W1 - lr * gW1, b1 - lr * gb1
    out = act(X @ W1 + b1) @ W2 + b2
    return float(np.mean((out - y) ** 2)), out.ravel(), W1.ravel()

for seed in (0, 1, 2):
    flat_loss, flat_out, _ = train(seed, nonlinear=False)
    tanh_loss, tanh_out, W1 = train(seed, nonlinear=True)
    print(f"seed {seed}  no activation  loss {flat_loss:.4f}  outputs {np.round(flat_out, 2)}")
    print(f"seed {seed}  tanh           loss {tanh_loss:.4f}  outputs {np.round(tanh_out, 2)}"
          f"  layer-1 weights {np.round(W1, 2)}")

solved = sum(train(s, nonlinear=True)[0] < 1e-3 for s in range(30))
print(f"\ntanh network solves XOR from {solved} of 30 random starts")
print(f"no-activation network solves it from "
      f"{sum(train(s, nonlinear=False)[0] < 1e-3 for s in range(30))} of 30")

Output:

seed 0  no activation  loss 0.2500  outputs [0.5 0.5 0.5 0.5]
seed 0  tanh           loss 0.0000  outputs [0. 1. 1. 0.]  layer-1 weights [0.81 1.71 0.81 1.74]
seed 1  no activation  loss 0.2500  outputs [0.5 0.5 0.5 0.5]
seed 1  tanh           loss 0.1251  outputs [0.  0.5 1.  0.5]  layer-1 weights [ 1.5  -0.08  3.83 -2.93]
seed 2  no activation  loss 0.2500  outputs [0.5 0.5 0.5 0.5]
seed 2  tanh           loss 0.0000  outputs [0. 1. 1. 0.]  layer-1 weights [ 1.58 -0.78  3.18 -0.95]

tanh network solves XOR from 28 of 30 random starts
no-activation network solves it from 0 of 30

Three things in that output are worth more than the code. Without the activation, every run lands on 0.2500 and predicts 0.5 for all four inputs — that is not a tuning failure, it is the best a straight line can do when two of the targets are 1 and two are 0, and no learning rate or step budget changes it. With tanh, the same nine parameters reach zero loss. And seed 1 does not: it settles at 0.1251, getting two of the four cases right and hedging on the rest, which is what a local minimum looks like from the inside. Two of thirty starts fail on a problem with four training examples. Scale that intuition up before assuming a model that underperforms must be too small.