Definition
A layer in a neural network is one rectangular grid of numbers — a weight matrix — plus a short list of biases and a nonlinear function applied to the result. A vector of numbers arrives, the layer multiplies it by the matrix, adds the biases, and pushes each output through something like ReLU. That is the entire object. Everything else in deep learning is a decision about how many of these to stack and what shape to make each one.
The three words a first tutorial runs together sit at three different scales. A dense layer with 512 inputs and 2,048 outputs has 2,048 neurons — one per output, each owning one column of the matrix — and 512 × 2,048 + 2,048 = 1,050,624 parameters, the individual numbers that training adjusts. A neuron is a slice of the layer's arithmetic; a parameter is a single cell inside it. When a model is described as having 175 billion parameters, nobody is counting layers or neurons. They are counting cells.
Layers exist as a concept because a stack of them can express things one of them cannot, and the nonlinearity is what makes that true. Two linear layers back to back are exactly equivalent to one: multiplying by W1 and then by W2 is multiplying by the single matrix W1 @ W2, so a hundred stacked linear layers have precisely the expressive power of one. Insert a ReLU between them and no single matrix reproduces the pair. Depth buys you nothing without an activation function, and buys you a great deal with one.
How It Works
Each layer is defined entirely by two shapes: how many numbers come in and how many go out. A layer that takes 512 and emits 2,048 owns a 512 × 2,048 matrix of weights and 2,048 biases. Because the output width of one layer is the input width of the next, a network is a chain of shape agreements, and "adding a layer" means inserting another matrix into that chain.
The base model in the original Transformer paper makes the arithmetic concrete. Vaswani et al. specify a model dimension of 512 and a feed-forward inner dimension of 2,048, so each feed-forward sub-layer is a 512 → 2,048 matrix followed by a 2,048 → 512 matrix: 1,050,624 + 1,049,088 = 2,099,712 parameters per block. The four attention projections in the same block are each 512 × 512, another 1,048,576. Multiply the roughly 3.15 million per layer by the paper's N = 6 encoder layers and the stack accounts for about 19 million of the base model's 65 million; embeddings and the decoder hold the rest.
Scale that identical structure up and it explains almost the whole of a modern model. GPT-3 175B has 96 layers with a model dimension of 12,288 and, per the paper, a feed-forward width of 4 × 12,288 = 49,152. Each layer therefore carries 4 × 12,288² = 603,979,776 attention parameters plus 2 × 12,288 × 49,152 = 1,207,959,552 feed-forward parameters, or 1.81 billion per layer. Ninety-six of those come to 173.9 billion — essentially the entire 175 billion. A large language model is not an elaborate structure. It is one modest block, copied ninety-six times, with an embedding table bolted on the front.
What the layers actually do to the data is a chain of representations. The first layer sees raw input and produces 2,048 numbers that mean nothing to a human; the second sees those numbers, not the input. In a vision network this compounds into a familiar hierarchy — oriented edges, then corners and textures, then object parts — because a unit in layer three can only see input through the window that layers one and two gave it. In a language model the same compounding turns token identities into something that encodes syntax, then reference, then task.
Training reverses the flow. Backpropagation computes how much each output was wrong, then walks the chain backwards, multiplying by each layer's transposed weight matrix to work out how much every parameter in every earlier layer contributed. This is why depth is a training problem and not only an architecture choice: the gradient reaching layer 1 in a 96-layer network has been through 95 multiplications, and if those multiplications systematically shrink or inflate the signal, the early layers receive either nothing or nonsense.
Types
The names below are the ones practitioners and papers actually use, and each describes a different operation on the tensor — not a different topic.
Dense (fully connected) layers
Every input connects to every output, so the parameter count is inputs × outputs + outputs and grows as the product of the two widths. Dense layers make no assumption about the structure of their input; a dense layer would learn equally well from an image whose pixels had been randomly shuffled, provided the same shuffle were applied every time. That generality is exactly why they are expensive.
Convolutional layers
A convolutional layer applies one small filter at every position of a grid, reusing the same weights everywhere instead of learning a separate weight per pixel. The saving is enormous. AlexNet's first convolutional layer holds 96 kernels of 11 × 11 × 3, which is 34,848 weights. A dense layer reading the same 224 × 224 RGB image — 150,528 numbers — and producing 4,096 outputs would need 616,562,688: about 17,700 times more parameters for a layer that also cannot recognise a shape it has only seen in the top-left corner. Krizhevsky measured the resulting lopsidedness in the network as a whole: convolutional layers "contain about 90-95% of the computation, about 5% of the parameters", while fully-connected layers hold about 95% of the parameters and 5-10% of the computation.
Recurrent layers
A recurrent layer has one weight matrix that it applies again at every time step, carrying a hidden state forward. Its depth in parameters is one layer; its depth in computation is the length of the sequence, which is why a 100-token sequence puts 100 multiplications between the first token's gradient and the loss. LSTM and GRU variants exist entirely to keep that long chain from collapsing.
Attention layers
Self-attention is the only common layer whose mixing pattern is computed from the data rather than fixed by the weights. A dense layer always combines its inputs the same way; an attention layer builds a fresh weighting for every input by comparing each position against every other, which is where its quadratic cost in sequence length comes from. Its learned parameters are just the four projection matrices that produce queries, keys, values and the output.
Normalization layers
Layer norm and batch norm do not change the shape of anything. They rescale a layer's outputs to a controlled mean and variance so that the next layer receives numbers in a predictable range. They are cheap and they are load-bearing: Ioffe and Szegedy reported batch normalization reaching the same accuracy as their Inception baseline "with 14 times fewer training steps", and training stably at thirty times the original learning rate — a five-fold increase alone had driven the un-normalized network's parameters "to machine infinity".
Embedding layers
An embedding layer is a lookup table, not a multiplication: one row per vocabulary entry, each row a vector of the model's width. It is usually the single largest weight matrix in a language model and does the least arithmetic, since selecting row 4,271 costs a memory read rather than a matmul.
Pooling and dropout layers
Neither holds a learnable parameter. Pooling shrinks a grid by taking the maximum or mean of each small window; dropout zeroes a random fraction of activations during training only. This is why papers count weight layers rather than layers: VGG-16's sixteen are the sixteen that hold parameters, and the pooling stages between them do not count.
Real-World Applications
AlexNet, ImageNet 2012. Five convolutional layers and roughly 60 million parameters won the competition that started the modern era, and its parameter distribution set the agenda for the next decade: nearly all the weight sat in the fully-connected layers at the end, nearly all the compute in the convolutional layers at the front. Every efficiency architecture since has been an attack on one of those two facts.
VGG, 2014. Simonyan and Zisserman built configurations "from 11 weight layers in the network A (8 conv. and 3 FC layers) to 19 weight layers in the network E (16 conv. and 3 FC layers)" on a fixed 224 × 224 RGB input, and reported parameter counts of 133M for the 11-layer version and 144M for the 19-layer one. Nearly doubling the depth changed the parameter count by 8%, because the dense classifier head dominated the total regardless. VGG is the cleanest evidence in the literature that layer count and model size are separate quantities.
ResNet, ILSVRC 2015. He et al. reported that a 56-layer plain network had higher training error than a 20-layer one on CIFAR-10 — and the same effect at 18 versus 34 layers on ImageNet. Adding a shortcut that carries the input around each pair of layers removed the problem; their 152-layer residual network reached 4.49% single-model top-5 validation error, an ensemble took 3.57% on the test set and first place, and a 1,202-layer variant trained on CIFAR-10 without optimisation collapse. Residual connections are now in essentially every deep architecture, including every Transformer.
The Transformer, 2017. Six encoder and six decoder layers, each with an attention sub-layer and a feed-forward sub-layer, each wrapped as LayerNorm(x + Sublayer(x)). That wrapper — residual first, then normalize — is the specific arrangement of layers that made stacks of ninety and more trainable.
GPT-3, 2020. Ninety-six identical layers at width 12,288. The comparison inside the same paper is the useful part: GPT-3 Small is 12 layers at width 768 and 125M parameters. Going to the full model multiplied depth by 8 and width by 16, and multiplied parameters by 1,400.
Key Concepts
Depth versus width. These are the two dimensions of a stack, and they do not cost the same. Adding a layer adds parameters linearly; widening every layer adds them quadratically, because a dense layer's size is the product of its two widths. That is why GPT-3's eight-fold increase in depth over GPT-3 Small accounts for so little of its fourteen-hundred-fold increase in parameters. Depth buys the number of successive transformations the network can apply; width buys how much information each one can carry.
Layer count is not capacity. VGG-19 has 73% more weight layers than VGG-11 and 8% more parameters. A 96-layer model with width 768 would be a small model. Any statement of the form "deeper means more powerful" that does not mention width is describing at most half the architecture.
Residual connections change what a layer has to learn. With a shortcut, a layer computes a correction to its input rather than a replacement for it, so the identity function — doing nothing — is available for free instead of having to be learned. This is what makes a very deep stack no worse than a shallow one, which is precisely the property the plain 56-layer network failed to have.
Not every layer holds weights. Pooling, dropout and standalone activation layers appear in a framework's layer list and contribute nothing to the parameter count. A model summary showing 60 layers may hold weights in half of them.
Challenges
Depth can make training error worse, and that is not overfitting. The single most useful result to know here is He et al.'s: a 56-layer plain convolutional network fits its training data worse than a 20-layer one. Overfitting would show up as a training/test gap; this shows up in training error itself, which means the optimiser cannot find a solution it demonstrably possesses — the shallower network's weights, padded with identity layers, are a valid setting of the deeper one. Reaching for dropout or more data when the training loss is the thing that is too high is a wasted week.
Every extra layer is another multiplication in the gradient chain. Backpropagation multiplies by each layer's weights on the way back, so if the typical factor is 0.9 the signal reaching a layer 50 steps from the loss is scaled by roughly 0.9⁵⁰ ≈ 0.005; if it is 1.1, the same 50 steps scale it by about 117. Neither is a training run. Normalization layers, careful initialisation and residual paths are all mitigations of this one compounding, which is why depth beyond about 20 layers was largely impractical before 2015 and routine after.
Depth costs training memory that inference does not pay. The forward pass of a 96-layer model can discard each layer's activations as it goes; the backward pass cannot, because computing a layer's gradient needs the input it saw. Training memory therefore grows with depth even though the weights are unchanged, and it grows with batch size at the same time. This is the reason activation checkpointing exists, and the reason a model that fits comfortably for inference can fail to fit for fine-tuning on the same hardware.
Layer widths propagate. Changing one layer's output width changes the next layer's input width, so a hidden size is not a local decision. In a Transformer it is worse than local: the model dimension appears in the attention projections, the feed-forward matrices, the embedding table and the residual stream at once, which is why architectures publish one d_model rather than a per-layer table.
Future Trends
Sparse layers decouple parameters from compute. A mixture-of-experts layer replaces the single feed-forward block with many, and routes each token to a small number of them. The layer's parameter count rises by the number of experts while the arithmetic performed per token stays close to the dense case — the first serious break in the assumption, implicit in every calculation above, that a layer's size and its cost are the same number.
Where the normalization sits inside the layer is itself a research question. The original Transformer normalized after the residual addition; nearly every large model since normalizes before it, because doing so makes the gradient at initialisation better behaved and removes the need for a learning-rate warmup schedule. Variants that normalize in both places, or that rescale the residual branch instead, are still being published — an argument about the internal order of three operations, at a scale where getting it wrong wastes a training run.
Layers are becoming a unit of compression. Because a residual layer computes a correction, some of them turn out to correct very little, and depth pruning — deleting whole layers from a trained model and briefly healing the result — is emerging alongside quantization and distillation as a way to shrink a model. It only works at all because of the residual structure: remove a layer from a non-residual stack and you break the chain of shape agreements and the learned function together.
Conditional depth. Early-exit and layer-skipping schemes let an easy input leave the stack after a fraction of the layers, making depth a property of the input rather than of the model. This turns "how many layers does this model have" into a question with a distribution for an answer, and complicates every serving system that assumed a fixed per-token cost.
Code Example
A layer is small enough to write in one line, and doing so makes both the parameter arithmetic and the reason for the nonlinearity concrete.
import numpy as np
rng = np.random.default_rng(0)
def dense(x, W, b):
"""One layer: a matrix multiply, a bias vector, and nothing else."""
return x @ W + b
# A layer is fixed entirely by its two shapes: 512 numbers in, 2048 numbers out.
W1, b1 = rng.normal(size=(512, 2048)) * 0.02, np.zeros(2048)
W2, b2 = rng.normal(size=(2048, 512)) * 0.02, np.zeros(512)
print("layer 1 parameters:", W1.size + b1.size)
print("layer 2 parameters:", W2.size + b2.size)
print("feed-forward block total:", W1.size + b1.size + W2.size + b2.size)
x = rng.normal(size=(1, 512))
# With no activation between them, the two layers collapse into a single matrix.
collapsed = x @ (W1 @ W2) + (b1 @ W2 + b2)
print("linear stack == one 512x512 layer:", np.allclose(dense(dense(x, W1, b1), W2, b2), collapsed))
# Add the nonlinearity and no single matrix can reproduce the pair.
relu = lambda t: np.maximum(t, 0.0)
print("ReLU stack == one 512x512 layer:", np.allclose(dense(relu(dense(x, W1, b1)), W2, b2), collapsed))
Output:
layer 1 parameters: 1050624
layer 2 parameters: 1049088
feed-forward block total: 2099712
linear stack == one 512x512 layer: True
ReLU stack == one 512x512 layer: False
Those 2,099,712 parameters are one feed-forward block of the Transformer base model, and the last two lines are the whole argument for depth: without relu, the second layer was free, in the sense that a single 512 × 512 matrix could have done its job. With relu, it is not.