Positional Encoding (RoPE)

Definition

Positional encoding is the information a transformer adds to each token so the model knows where that token sits in the sequence. It is needed because the core operation of a transformer — self-attention — is permutation-invariant: it treats its input as an unordered set and produces the same output no matter what order the tokens come in. Add nothing, and the model cannot distinguish "dog bites man" from "man bites dog," because to the attention math the two are the same bag of three words.

That single fact is the whole reason the concept exists, and it is easy to miss because word order is so obviously meaningful to us. Attention computes, for every pair of tokens, a score from their content vectors and then mixes them together — but nowhere in that computation does the index of a token appear. Shuffle the rows of the input and you shuffle the rows of the output identically; the numbers inside do not change. Position has to be injected deliberately, and the design choices for how to inject it are what this page is about. The current default in large language models is RoPE (Rotary Position Embedding), which encodes position by rotating each token's query and key vectors — a scheme that, as a side effect, makes attention depend on how far apart two tokens are rather than on where they sit.

How It Works

Why position must be injected

A transformer turns each token into a vector — its embedding — and then lets self-attention mix those vectors together. Attention builds each token's output as a weighted sum of all the tokens' value vectors, where the weights come from dot products between query and key vectors. Nothing in that arithmetic references a token's index. This is what people mean when they say attention is a "set operation": feed it the same tokens in a different order and every output permutes to match, but no output value changes. The position signal is the only thing that tells the model that the second word is the second word.

The fix, in every scheme below, is to modify the token vectors so that identical content at different positions produces different numbers. The schemes differ in what they modify and when.

Absolute sinusoidal encoding — the original recipe

The 2017 paper "Attention Is All You Need" added a fixed pattern to each embedding before the first layer. For position pos and embedding dimension i, it defines PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) for the even dimensions and cos for the odd ones, where d_model is the embedding width (512 in the base model). Each dimension is a sine wave, and the wavelengths form a geometric progression from 2π up to 10000 · 2π — so low dimensions oscillate quickly (distinguishing neighbours) and high dimensions turn slowly (distinguishing far-apart regions). The transformer page works through this formula in more detail; the key point here is that the base constant 10000 sets the range of wavelengths, and the authors chose sinusoids partly because "it may allow the model to extrapolate to sequence lengths longer than the ones encountered during training." That hope turned out to be optimistic, which is much of the story below.

RoPE — encoding position as a rotation

RoPE, introduced in the 2021 RoFormer paper "RoFormer: Enhanced Transformer with Rotary Position Embedding", takes a different route. Instead of adding a position vector, it rotates the query and key vectors by an angle proportional to their position. Split a query vector into 2-D pairs; the pair i at position m is rotated by the angle m · θ_i, where the per-pair frequencies θ_i reuse the same 10000^(-2i/d) schedule as the sinusoidal encoding. The keys are rotated the same way by their own positions.

The reason this is clever is a fact from plane geometry: the dot product of two rotated vectors depends only on the difference of their rotation angles. If a query at position m is rotated by m·θ and a key at position n by n·θ, their dot product is a function of (m − n) alone. The RoFormer paper states this as its design goal — that the inner product satisfies ⟨f_q(x_m, m), f_k(x_n, n)⟩ = g(x_m, x_n, m − n), a function of the two content vectors and their relative distance. So RoPE applies an absolute rotation to each token, yet the attention score it produces is inherently relative.

A worked example makes the invariance concrete. Take a query and a key that are both the 2-D vector (1, 0), and a rotation frequency of 30° per position. Put them one token apart:

• At positions 2 and 3: rotate the query by 2 × 30° = 60°, giving (0.5, 0.866), and the key by 3 × 30° = 90°, giving (0, 1). Their dot product is 0.5 × 0 + 0.866 × 1 = 0.866.
• At positions 7 and 8: rotate the query by 210°, giving (−0.866, −0.5), and the key by 240°, giving (−0.5, −0.866). Their dot product is (−0.866)(−0.5) + (−0.5)(−0.866) = 0.866.

Same answer, 0.866 = cos(30°), because both pairs are one position apart. The absolute positions moved from 2–3 to 7–8; the attention score did not. That is exactly the property you want in a language model: how much "the" attends to the noun after it should not depend on whether the phrase is at the start or the end of the document. The Code Example reproduces this on full-width vectors and confirms the numbers.

Types

There is a genuine four-way taxonomy here — these are names researchers actually use, not categories invented to fill a section.

• Sinusoidal absolute (the original transformer): a fixed, non-learned pattern added to each embedding. No parameters, deterministic at any length, but the model only ever sees the patterns for positions inside its training range.
• Learned absolute (BERT, GPT-2, the original vision transformer): one trainable embedding per position, looked up and added like a token embedding. Flexible during training, but it has a hard maximum length — there is no learned vector for position 4,097 if you only trained 4,096 — so it cannot extend at all without new parameters.
• Relative (Shaw et al. 2018; the T5 family): instead of a per-position signal, a learned bias added to the attention score based on the distance between the two tokens. This bakes translation-invariance in directly, at the cost of a more intrusive change to the attention computation.
• Rotary / RoPE: the relative behaviour of a distance-based scheme, obtained through absolute rotations of the query and key vectors. It needs no extra parameters, composes cleanly with efficient attention kernels, and — crucially — its behaviour can be retuned after training by changing the rotation frequencies. That last property is why it dominates today.

A fifth option worth naming is no positional encoding at all: a causal decoder can partially infer position from the attention mask itself, and some recent long-context designs deliberately interleave layers that carry no position signal with RoPE layers. It is the exception that proves the rule — the mask is itself a source of order information.

Real-World Applications

RoPE is the positional scheme in the large majority of open large language models. Meta's LLaMA and Llama 2/3 families, EleutherAI's GPT-NeoX, Google's PaLM, Mistral's models and Qwen all use it; it is close to a default in decoder-only architectures. Learned absolute encoding is what the earlier generation used — BERT and GPT-2 both carry a fixed table of learned position vectors — and it remains common in encoders and in vision transformers, where the sequence length (the number of image patches) is fixed by design and extrapolation is not needed.

The most consequential application of RoPE specifically is context-window extension: taking a model trained at one sequence length and making it work at several times that length without retraining from scratch. Three techniques, all from 2023, do this by manipulating RoPE's rotation frequencies:

• Position Interpolation (Chen et al., Meta, June 2023) linearly down-scales position indices so they stay inside the trained range — see the worked calculation below. It extended LLaMA models to 32,768 tokens with fine-tuning within roughly a thousand steps.
• NTK-aware interpolation (a community proposal from the same period) instead changes the base constant so that high-frequency dimensions, which carry fine local information, lose less of it during the rescaling.
• YaRN ("Yet another RoPE extensioN," Peng et al., 2023) combined a refined interpolation with a rescaling of the attention logits, reaching context windows around 128K tokens far more cheaply than earlier methods.

These figures are stated as of the 2023 papers that introduced them; production models have since pushed much further. Long-context designs also mix schemes — as documented on the context window page, one route to a very large window interleaves attention layers that carry no positional embeddings with rotary-embedded ones. The relevant point for this page is durable: because RoPE's positions live in adjustable frequencies rather than a fixed lookup table, it is the scheme long-context work is built on.

Key Concepts

Relative versus absolute

The distinction that organises the whole topic is absolute (this token is at index 5) versus relative (these two tokens are 3 apart). Language cares about the relative version far more: a subject-verb agreement or a pronoun reference depends on the gap between words, not on their offset from the document's start. RoPE's appeal is that it delivers relative behaviour — attention scores that depend on (m − n) — while only ever applying a per-token absolute operation, which is cheap and easy to slot into existing attention kernels.

The base frequency and why it controls long context

The base constant (10000 in the original recipe) fixes how fast the slowest rotation turns, and therefore the longest distance the model can represent before the rotation angles wrap around and start repeating. This is why the base is the knob every extension method reaches for. Position Interpolation shows the idea at its simplest: to run a model trained on 2,048 tokens at 8,192 tokens, multiply every position index by 2048 / 8192 = 0.25. Position 8,000 is then fed to the model as position 2,000 — inside the range it was trained on — instead of as an unseen 8,000 that would produce rotation angles the model has never encountered. A short fine-tune teaches the model to read the finer-grained fractional positions, and its usable window quadruples. The trade is resolution: you have packed four times as many positions into the same angular space, so neighbouring tokens are harder to tell apart, which is what NTK-aware and YaRN refinements try to recover.

Challenges

The extrapolation cliff. A model trained at one length does not automatically work at a longer one, and this is a property of the position signal specifically. Absolute learned encodings fail hardest — there is simply no vector for a position past the training maximum. Sinusoidal and RoPE encodings are defined at any length, so they do not crash, but run well past the training range and quality still collapses: the rotation angles fall outside the distribution the attention layers learned to interpret. "Train short, run long" is an engineering project, not a free parameter.

Retuning is fiddly and lossy. Every extension method trades something. Naive interpolation blurs local resolution; changing the base to protect high frequencies can distort long-range behaviour; YaRN needed an extra attention-temperature correction precisely because rescaling positions quietly changes the scale of the attention logits. There is no single setting that is best at all lengths, so long-context models are tuned for a target window rather than made length-agnostic.

RoPE's own long-term decay can work against recall. The RoFormer paper presents decreasing attention with distance as a feature — distant tokens should influence each other less. For a retrieval-style task where the answer sits 100,000 tokens back, that built-in decay is working against you, and it is part of why raw context length and usable context length diverge so sharply.

It interacts with everything downstream. Because RoPE modifies queries and keys, it has to be applied inside the attention kernel, so it must be kept compatible with the fused-attention implementations (such as FlashAttention) and the memory-saving attention variants (such as grouped-query attention) that production inference depends on. A positional scheme that could not compose with those would be a non-starter regardless of its accuracy.

Future Trends

The durable open problem is length generalisation: making a model that trains cheaply on short sequences and still reasons correctly on much longer ones, without a bespoke extension recipe per target length. Research is pushing in two directions at once — smarter frequency schedules that extrapolate further before degrading, and the surprising finding that causal decoders can infer a good deal of order information from the attention mask alone, which has revived interest in reducing or removing explicit positional signals in some layers. Meanwhile, architectures outside the transformer family, such as state space models, handle position intrinsically through their recurrence and sidestep the question entirely; how much of the long-context problem is really a positional-encoding problem is, as of this writing, genuinely unsettled.

Code Example

This reproduces the worked example on full-width vectors. It rotates a query and a key with RoPE and computes their attention score at several absolute positions that share the same relative distance. Run as shown, it prints identical scores for equal distances and a different score when the distance changes — the relative-position property, on numbers.

import numpy as np

def rope_rotate(x, pos, base=10000):
    # Rotate each adjacent 2D pair of x by pos * theta_i.
    d = len(x)
    out = np.empty(d, dtype=float)
    for i in range(d // 2):
        theta = base ** (-2 * i / d)          # frequency for pair i
        angle = pos * theta
        c, s = np.cos(angle), np.sin(angle)
        a, b = x[2 * i], x[2 * i + 1]
        out[2 * i]     = a * c - b * s
        out[2 * i + 1] = a * s + b * c
    return out

q = np.array([0.9, 0.1, -0.4, 0.7])   # a query vector
k = np.array([0.2, 0.8,  0.5, 0.3])   # a key vector

# Same relative distance (3), different absolute positions:
for m, n in [(2, 5), (10, 13), (50, 53)]:
    score = rope_rotate(q, m) @ rope_rotate(k, n)
    print(f"positions ({m:>2},{n:>2}), distance {n - m}: score = {score:.6f}")

# A different distance gives a different score:
score = rope_rotate(q, 2) @ rope_rotate(k, 4)
print(f"positions ( 2, 4), distance 2: score = {score:.6f}")

Output:

positions ( 2, 5), distance 3: score = -0.332089
positions (10,13), distance 3: score = -0.332089
positions (50,53), distance 3: score = -0.332089
positions ( 2, 4), distance 2: score = -0.725309

The score for distance 3 is the same whether the tokens sit at positions 2–5, 10–13, or 50–53 — the absolute positions are invisible to the attention score, exactly as the RoFormer derivation requires. Change the distance to 2 and the score changes. That translation-invariance, obtained for free from a per-token rotation, is the whole reason RoPE displaced the schemes that came before it.