Recurrent Neural Network (RNN)

Definition

A recurrent neural network (RNN) is a neural network that processes a sequence one element at a time while carrying a hidden state — a fixed-size vector that summarizes everything it has read so far. At each step it combines the current input with the hidden state left over from the previous step, produces a new hidden state, and passes that forward. This loop, feeding the previous output of the layer back into itself, is the "recurrent" part, and it is what lets the network use earlier context to interpret what comes next.

That single idea is the whole difference from an ordinary feedforward network, which treats every input in isolation and has no memory of what came before. Because word order, note order, and the timing of events carry meaning, the hidden state is what makes an RNN suited to text, speech, and time series. The catch — the reason architectures like the LSTM exist — is that the same loop makes RNNs hard to train over long spans, which is the story of the rest of this page.

How It Works

Unroll the loop and an RNN is easier to see. Give it a sequence of inputs x₁, x₂, x₃, and so on. It starts with an empty hidden state h₀ (usually all zeros). At each step t it computes a new hidden state from two things: the current input xₜ and the previous hidden state hₜ₋₁. In its simplest form:

hₜ = tanh(Wₓ · xₜ + Wₕ · hₜ₋₁ + b)

Wₓ, Wₕ, and b are learned weights, and tanh squashes the result into a bounded range. The key detail is that these weights are the same at every step — the network does not have separate parameters for position 1, position 2, and so on. It applies one small set of rules over and over, and all the memory of the past lives in hₜ. An output can be read off the hidden state at every step (as in tagging each word) or only at the end (as in classifying a whole sentence).

Training uses backpropagation through time (BPTT): the unrolled network is treated like a deep feedforward network that happens to be one layer repeated, and the error is propagated backward from the final step all the way to the first. A 40-word sentence unrolls into a 40-layer-deep computation. That depth is exactly where the trouble starts, because the gradient reaching step 1 is a product of one factor per step it passed through — and a long product of numbers is either negligibly small or enormous. This is the vanishing and exploding gradient problem, covered under Challenges.

Types

The useful taxonomy of RNNs is the design of the recurrent cell itself. All three below reuse one set of weights across time; they differ in how carefully they manage the hidden state.

Simple (vanilla) RNN

The plain version described above: one hidden state, one tanh update, no special machinery. It is easy to understand and cheap to run, and it works when the relevant context is a few steps back. It struggles badly when the answer depends on something dozens of steps earlier, because the learning signal for that distant input vanishes before it arrives. In practice a vanilla RNN rarely learns dependencies longer than roughly 10 steps.

Long Short-Term Memory (LSTM)

The LSTM, introduced by Hochreiter and Schmidhuber in 1997, was designed specifically to defeat the vanishing gradient. It adds a second running vector, the cell state, that flows through time along a nearly linear path — information can travel along it almost unchanged, so a gradient can survive many steps instead of being multiplied down to nothing. Access to that cell is controlled by learned gates, each a small layer that outputs values between 0 (block) and 1 (pass):

• The input gate decides how much of the new candidate information to write into the cell.
• The forget gate decides how much of the existing cell contents to keep versus erase.
• The output gate decides how much of the cell to expose as the visible hidden state.

Because the gates are learned, the network works out for itself what is worth remembering and for how long. (Historical note: the original 1997 design had only input and output gates; the forget gate that lets a cell reset itself was added by Gers, Schmidhuber, and Cummins in 2000, and is now standard.)

Gated Recurrent Unit (GRU)

The GRU is a streamlined gated cell. It merges the cell and hidden state into one vector and uses two gates instead of three — an update gate (how much of the past state to carry forward versus refresh) and a reset gate (how much past state to ignore when forming the new candidate). With fewer gates it has fewer parameters and trains faster than an LSTM, and on many tasks the two perform comparably, so which to prefer is usually settled empirically rather than in principle.

Real-World Applications

Before transformers took over, RNNs — almost always LSTMs or GRUs in practice — powered a wave of production sequence systems, and several of those uses remain live where the deployment favors streaming or small models.

The clearest large-scale example is machine translation: Google's Neural Machine Translation system, deployed in 2016, used deep stacked LSTMs for both the encoder and decoder and replaced the company's older phrase-based translator. Speech recognition was another stronghold — LSTM acoustic models were deployed in Google's voice systems in the mid-2010s, reading a stream of audio frames and emitting phonemes, a naturally sequential, real-time job. Alex Graves' work on LSTM handwriting recognition and synthesis showed the same architecture generating cursive one stroke at a time.

RNNs still fit where inputs arrive as an unbounded stream and latency matters: on-device keyboard prediction and gesture typing, sensor and control loops on embedded hardware, and financial or operational time-series forecasting where sequences are long but each model is small. The honest framing is that for large-scale language work this is now history — see the note below on why transformers won.

Key Concepts

• Hidden state: the fixed-size vector passed from each step to the next; the network's entire memory of the sequence so far is compressed into it.
• Parameter sharing across time: one set of weights is reused at every position, which is what lets an RNN handle sequences of any length and generalize a pattern regardless of where it occurs.
• Backpropagation through time (BPTT): training by unrolling the loop into a deep chain and propagating error from the last step back to the first.
• Gradient clipping: a standard fix for exploding gradients — rescale any gradient whose magnitude exceeds a threshold. It does nothing for vanishing gradients, which is why gated cells, not clipping, are the answer there.
• Bidirectional wiring: running one RNN forward and another backward over the same sequence and combining them, so each position sees both past and future context. This is orthogonal to the cell type — you can make a simple RNN, LSTM, or GRU bidirectional.

Challenges

The defining difficulty of RNNs is the vanishing and exploding gradient problem, and it is worth seeing on numbers because it explains every design decision above. When BPTT propagates error back through a long sequence, the gradient that reaches an early step is a product of roughly one factor per step in between. Suppose that factor averages 0.5. Over 20 steps the gradient is scaled by 0.5²⁰, which is about 0.00000095 — under one part in a million. The signal telling step 1 how to change has effectively disappeared, so the network cannot learn that a word 20 positions back mattered. Push the factor slightly above 1 instead, say 1.5, and 1.5²⁰ is about 3,325: the gradient explodes, updates overshoot wildly, and training diverges. Bengio, Simard, and Frasconi analyzed exactly this trade-off in 1994, and the LSTM was the answer to it in 1997. Gated cells and gradient clipping tame both failure modes but do not abolish them; effective memory in practice is long but not unlimited.

The second challenge is structural and is why RNNs lost the language crown: recurrence is inherently sequential. Step t cannot be computed until step t−1 is done, so an RNN cannot spread the work across a sequence in parallel the way a transformer can. On modern hardware built for parallel throughput, that is a decisive disadvantage — a transformer processes all positions at once and uses self-attention to connect any two positions directly, sidestepping the long multiplicative chain that gives RNNs their gradient problem in the first place. That combination is why transformers have largely displaced RNNs across natural language processing and most other large-scale sequence tasks, and why RNNs today live mainly in streaming and resource-constrained corners rather than at the frontier — though state-space models have since rebuilt a linear form of recurrence that trains in parallel, which is why the two accounts of "recurrence" do not actually conflict.

Code Example

A minimal vanilla RNN cell, run forward over a five-step sequence, followed by the gradient-factor arithmetic that shows why long-range learning breaks:

import numpy as np

# h_t = tanh(W_x . x_t + W_h . h_{t-1} + b)
np.random.seed(0)
hidden = 4
W_x = np.random.randn(hidden, 3) * 0.1
W_h = np.random.randn(hidden, hidden) * 0.1
b   = np.zeros(hidden)

sequence = [np.random.randn(3) for _ in range(5)]   # 5 time steps
h = np.zeros(hidden)                                 # initial hidden state
for t, x in enumerate(sequence):
    h = np.tanh(W_x @ x + W_h @ h + b)               # reuse the SAME weights each step
    print(f"step {t}: hidden state = {np.round(h, 3)}")

# Why long-range learning breaks: backprop multiplies one factor per step.
for g in (0.5, 1.5):
    print(f"gradient factor {g} over 20 steps -> {g**20:.2e}")

Output:

step 0: hidden state = [0.331 0.539 0.121 0.106]
step 1: hidden state = [-0.121  0.145  0.023 -0.282]
step 2: hidden state = [ 0.049 -0.164  0.015  0.123]
step 3: hidden state = [0.172 0.224 0.092 0.033]
step 4: hidden state = [-0.369 -0.302 -0.084 -0.296]
gradient factor 0.5 over 20 steps -> 9.54e-07
gradient factor 1.5 over 20 steps -> 3.33e+03

The loop reuses the same W_x and W_h at every step — that is the recurrence. The final two lines are the whole vanishing/exploding story: a per-step factor slightly under 1 collapses to a millionth over 20 steps, and a factor slightly over 1 blows up past three thousand.