Definition
A layer in a neural network does one thing: it multiplies its input by a matrix of weights and adds a bias. That is a linear operation, and linear operations compose into linear operations — stack two of them and you get W₂(W₁x) = (W₂W₁)x, which is just one matrix. A 100-layer network built only from such layers has exactly the representational power of a 1-layer one, however many billions of parameters it holds. An activation function is the small non-linear function applied to each neuron's output that breaks this collapse. It is the only reason depth buys anything at all.
Because one sits between every pair of layers, its shape also decides whether the network can be trained at all — the reason deep learning stayed impractical for two decades.
Which one should you use? In hidden layers, a ReLU-family function is the default: plain ReLU for convolutional and small dense networks, GELU inside a transformer feed-forward block, and a gated variant such as SwiGLU in recent large language models. Sigmoid and tanh belong at a bounded output or inside an LSTM gate, not in the middle of a deep stack.
How It Works
Each neuron computes a weighted sum z = Σ(wᵢxᵢ) + b and emits a = f(z). The forward pass is unremarkable; the interesting half is the backward pass.
Backpropagation carries the loss gradient from the output back toward the input by the chain rule, which multiplies by f′(z) exactly once per layer. The gradient reaching an early layer's weights is therefore a product of n activation derivatives, and the entire design question is: what does that number do when you multiply it by itself n times?
For the logistic sigmoid the derivative is σ(z)(1 − σ(z)). It peaks at 0.25 — at z = 0, the single most favourable point — and falls toward zero in both directions. Take that best case at every layer: after 10 layers the gradient has been scaled by 0.25¹⁰ ≈ 9.5 × 10⁻⁷, under one part in a million, and after 20 layers by roughly 9 × 10⁻¹³. Real pre-activations are not all parked at zero, so the true factor is worse. That is the vanishing-gradient problem with its arithmetic attached: early layers receive a gradient so small that gradient descent never meaningfully moves them.
ReLU is max(0, x), and its derivative is exactly 1 wherever the input is positive. Along any path of active units the gradient is multiplied by 1 at every layer, so depth costs it nothing: 1ⁿ = 1 for any n. It is also a comparison rather than an exponential — the 2012 AlexNet paper reported a four-layer convolutional network with ReLUs reaching 25% training error on CIFAR-10 about six times faster than the same network with tanh units. No gradient decay plus near-free evaluation is why "Deep Sparse Rectifier Neural Networks" displaced the S-curves, and why 100-plus-layer networks became trainable at all.
Types
Three families are in genuine use, and a fourth function is routinely misfiled with them.
Saturating S-curves — sigmoid and tanh. Both squash an unbounded input into a fixed range, 0 to 1 for sigmoid and −1 to 1 for tanh, and both flatten at the extremes, which is exactly where the derivative dies. Tanh is the better of the pair: it is zero-centred, and its derivative peaks at 1.0 rather than 0.25, decaying four times more slowly per layer. Neither is a sensible default today, but sigmoid survives wherever the bounded output is the point.
Rectifiers — ReLU and its variants. ReLU's flat negative half is simultaneously its strength and its defect: a unit pushed permanently negative outputs zero and receives zero gradient forever. Leaky ReLU replaces the flat half with a shallow slope, typically max(0.01x, x), leaving a dead unit a path back to life; PReLU learns that slope per channel. The variants deliver less than their popularity suggests; plain ReLU remains the pragmatic choice outside transformers.
Smooth and gated functions — GELU, SiLU/Swish, SwiGLU. GELU multiplies the input by the probability that a standard normal draw falls below it, x·Φ(x); SiLU/Swish uses x·σ(x). Both resemble ReLU from a distance but are differentiable everywhere and permit a small negative output near the origin; GELU became the transformer default through BERT and GPT-2. Gated linear units go further: the feed-forward block computes two projections of the same input and multiplies them element-wise, one gating the other. SwiGLU and GeGLU therefore need three weight matrices where a plain block needs two — 50% more parameters at equal hidden width — so implementations shrink the width to pay for it. Llama sets its feed-forward hidden dimension to (8/3)·d instead of the conventional 4·d, holding the parameter count level, and keeps the gate because it measurably helps.
Softmax is not an activation function in this sense. It is a normalisation over a vector: it exponentiates every element and divides by the sum, so the outputs are positive and total 1.0. Applied to a single neuron it means nothing. It lives at an output head, inside self-attention, and in mixture-of-experts routing — never between hidden layers.
Real-World Applications
The clearest case is ResNet (He et al., 2015), which trained a 152-layer image classifier using ReLU together with He initialisation — a weight variance of 2/fan_in, chosen because ReLU zeroes roughly half its inputs and so halves the variance a layer passes on. The activation and the initialisation are one design decision, not two.
In language models the split is generational. BERT and GPT-2 put GELU in every feed-forward block; newer open-weight families such as Llama and PaLM use SwiGLU instead, following "GLU Variants Improve Transformer" (Shazeer, 2020). Softmax appears in the same models twice over: across attention scores, and in the router that scores experts in a mixture-of-experts layer.
Recurrent models still use the S-curves deliberately: an LSTM applies sigmoid to its gates because a gate must be a multiplier in [0, 1]. At the deployment end the binding constraint is precision, not accuracy — MobileNet uses ReLU6, ReLU clipped at 6, because a bounded range survives fixed-point quantization, where an unbounded one lets one outlier stretch the INT8 scale and wreck precision for every other value.
Key Concepts
- Universal approximation: Cybenko (1989) and Hornik (1991) showed that one hidden layer with a non-polynomial activation can fit any continuous function on a compact domain to arbitrary accuracy. The theorem is silent on how wide that layer must be, which is why depth is what makes the result affordable.
- Saturation: sigmoid at z = 6 outputs 0.9975 with a derivative near 0.0025, so a unit that lands there is frozen until its inputs move substantially.
- Half the units emit zero: with roughly symmetric pre-activations, about 50% of a ReLU layer outputs exact zeros — free compression, and increasingly an inference optimisation.
- Zero-centring: when every output is positive (sigmoid, ReLU), every weight in the next layer receives a gradient of the same sign, forcing zig-zag update paths. Tanh, GELU and the gated variants avoid it.
- He and Xavier are not interchangeable: He initialisation assumes ReLU, Xavier/Glorot a symmetric S-curve. Mismatch them and activation variance shrinks or explodes layer by layer.
Challenges
Dying ReLU is the failure mode with real cost, and it is worse than it sounds because the gradient is exactly zero rather than merely small. A learning rate high enough to drive a unit's bias strongly negative kills it, and no later update can revive it — if 20% of a layer's units die, that layer's effective width is permanently 20% smaller for the rest of training. Lowering the learning rate prevents the problem; nothing fixes it afterwards.
ReLU is also not differentiable at zero; frameworks define the derivative there as 0 by convention, which is harmless in floating point but is a convention rather than a derivative. The smooth alternatives remove the kink at a price: GELU's exact form needs an error function, so most implementations ship a tanh approximation.
The deeper difficulty is that an activation cannot be swapped in isolation. Activation, initialisation, learning rate and normalisation are tuned as a set, so replacing GELU with ReLU in a trained architecture means retraining, not editing one line. And the honest summary of a decade of comparisons is deflationary: reported gains between ReLU-family functions are frequently inside seed-to-seed variance. The large win — from saturating to non-saturating functions — was collected once, around 2011, and has not been repeated.
Future Trends
Automated search produced Swish, and nothing found since has displaced the incumbents. Gating is the trend that stuck: SwiGLU went from a 2020 paper to the standard feed-forward activation of open-weight LLMs in about three years.
The live research direction inverts the last decade's story. Because ReLU zeroes roughly half its inputs, the rows of a feed-forward down-projection that a zero can never influence may simply be skipped — work such as "ReLU Strikes Back" (Mirzadeh et al., 2023) proposes putting ReLU back into LLM feed-forward blocks to buy exactly that inference-time sparsity, trading a little quality for memory traffic. Quantization pressure points the same way. Meanwhile layer normalisation and residual connections have absorbed much of the training instability that made the choice consequential, so for most practitioners the activation function is now a default rather than a decision.
Code Example
import numpy as np
def sigmoid(z):
return 1.0 / (1.0 + np.exp(-z))
# Backpropagation multiplies by the activation's derivative once per layer.
# The product over n layers is the whole story.
z = np.linspace(-6.0, 6.0, 13)
sig_deriv = sigmoid(z) * (1.0 - sigmoid(z)) # peaks at 0.25, at z = 0
relu_deriv = (z > 0).astype(float) # exactly 1 on the active side
print("best-case sigmoid derivative:", sig_deriv.max().round(4)) # 0.25
print("relu derivative when active: ", relu_deriv.max()) # 1.0
g_sig, g_relu = 1.0, 1.0
for layer in range(1, 21):
g_sig *= sig_deriv.max() # the most generous value sigmoid can ever supply
g_relu *= 1.0
if layer in (5, 10, 20):
print(f"{layer:2d} layers: sigmoid {g_sig:.2e} relu {g_relu:.2f}")
# 5 layers: sigmoid 9.77e-04 relu 1.00
# 10 layers: sigmoid 9.54e-07 relu 1.00
# 20 layers: sigmoid 9.09e-13 relu 1.00
The sigmoid column is a ceiling, not an estimate — it assumes every unit in every layer sits at the one input where the derivative is largest. A real network does considerably worse.