Definition
Self-attention is the operation that lets every token in a sequence look at every other token and decide, for each one, how much it matters for interpreting the current token. Each token gathers information from all the others as a weighted sum, where the weights are learned measures of relevance — so the representation of the word "it" can pull in the word "animal" three positions back if that is what "it" refers to. This is the core computation inside a transformer, and it is the reason large language models handle long-range context that recurrent neural networks could not.
The mechanism was introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. The word "self" is the important part: a recurrent network reads a sentence left to right and has to squeeze everything it has seen so far into one hidden state, so a dependency between the first and last word must survive dozens of sequential steps. Self-attention connects those two positions directly, in a single operation, no matter how far apart they sit. It generalises the broader idea of an attention mechanism to the case where a sequence attends to itself rather than to some other sequence.
How It Works
Every token starts as a vector (its embedding plus a position signal). Self-attention turns each of those vectors into three new vectors by multiplying it with three learned weight matrices:
- Query (Q) — what this token is looking for.
- Key (K) — what this token offers to others that are looking.
- Value (V) — the information this token actually passes on if it is attended to.
The intuition is a soft lookup. A token's Query is compared against the Key of every token (including itself); wherever a Query and a Key point in a similar direction, the dot product between them is large, meaning "this token is relevant to me." Those raw relevance scores are turned into weights that sum to one with a softmax, and the token's output is the weighted sum of everyone's Values. A token that scores 0.9 on one neighbour and near zero elsewhere has, in effect, copied that neighbour's Value; a token that spreads its weight evenly has averaged the whole sequence.
Packing all the tokens' Query, Key and Value vectors into matrices Q, K and V, the entire operation for the whole sequence at once is a single formula. Vaswani et al. call it Scaled Dot-Product Attention:
Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V
Read it left to right: QKᵀ produces an n×n grid of raw scores — one number for every ordered pair of tokens; dividing by √d_k rescales them (explained below); the softmax turns each row into a probability distribution over the sequence; and multiplying by V replaces each token with the relevance-weighted blend of every token's Value. In the base transformer the key dimension is d_k = 64 and the model dimension is d_model = 512, split across h = 8 attention heads that each run this computation on a different learned projection and are then concatenated.
A tiny worked example
Take the four-token fragment "The animal crossed road" and give each token an embedding. After projecting to Q, K and V and running the formula, the softmax produces a 4×4 grid of weights — one row per token, each row summing to 1. To see what a single row means, imagine the row for "crossed" came out as:
The animal crossed road
0.05 0.55 0.10 0.30
That row says "crossed" drew most of its information from "animal," some from "road," and almost none from "The." Its new representation is then 0.05·V_The + 0.55·V_animal + 0.10·V_crossed + 0.30·V_road — a vector that now blends in what the subject and object were. In a trained model those weights line up with grammatical and semantic relationships like this; in the untrained, random-weight run in the ## Code Example below they do not mean anything yet, but the machinery — a real 4×4 matrix whose rows each sum to 1 — is exactly the same.
Why divide by √d_k
The scaling is not cosmetic. If the components of a Query and a Key are independent with unit variance, their dot product has mean 0 and variance d_k — the raw scores grow with the dimension. Vaswani et al. note that "for large values of d_k, the dot products grow large in magnitude, pushing the softmax function into regions where it has extremely small gradients." A saturated softmax puts almost all its weight on one entry and returns near-zero gradients everywhere else, so learning stalls. Dividing by √d_k pulls the variance back to about 1. With d_k = 64 the divisor is simply √64 = 8.
Self-attention versus cross-attention
The formula is identical; the only difference is where Q, K and V come from. In self-attention all three are projections of the same sequence, so the sequence attends to itself — this is what stacks inside every layer of GPT- and BERT-style models. In cross-attention the Queries come from one sequence and the Keys and Values from another: a translation decoder generating French produces Queries that attend over the Keys and Values of the encoded English sentence. When people say "attention is what makes transformers work," the self-attention layers are the ones doing the heavy lifting on a single input.
Real-World Applications
Self-attention is not an optional add-on to these systems — it is the layer they are built from, repeated dozens of times.
- Decoder-only language models (GPT, Llama, Claude). These stack causal self-attention, where a mask blocks each token from attending to positions that come after it, so the model can only use the past when predicting the next token. This is what makes autoregressive text generation possible.
- Bidirectional encoders (BERT). The same self-attention with no mask, so every token sees the whole sentence at once — used for classification, retrieval and embeddings rather than generation.
- Machine translation, the original 2017 use case: encoder self-attention reads the source sentence, and the decoder combines causal self-attention over the output-so-far with cross-attention back into the source.
- Vision transformers (ViT) cut an image into patches and run self-attention over the patches as if they were tokens, letting a patch in one corner attend to a patch in the other — the mechanism that brought transformers into image classification.
Challenges
The O(n²) cost is the defining problem. Because every token attends to every other token, the score matrix QKᵀ has n² entries. For a sequence of n = 1,000 tokens that is 1,000 × 1,000 = 1,000,000 attention scores; at n = 10,000 it is 10,000 × 10,000 = 100,000,000. Ten times the length costs a hundred times the attention compute and, in a naive implementation, a hundred times the memory to hold the matrix. This single fact is why long context is expensive, why doubling a model's context window more than doubles the cost of using it, and why an entire research direction exists to get around it:
- FlashAttention keeps the exact same result but never materialises the full n×n matrix in slow memory, computing it in tiles and cutting the memory cost from quadratic to linear — a systems fix, not an approximation.
- Sparse and windowed attention change the math: each token attends only to a subset of positions (a local window, or a strided pattern) so the cost grows nearer to linear, at the price of some long-range links.
- The KV cache attacks the generation-time version of the cost: during autoregressive decoding the Keys and Values of earlier tokens never change, so they are stored and reused instead of recomputed — trading memory for compute, and turning the KV cache itself into the main memory consumer at long context.
A second, subtler challenge is interpretability. Attention weights are visible and tempting to read as explanations — "the model looked here" — but a large weight is not proof that a token was causally important to the output, and several studies have shown attention maps can be altered without changing predictions. Treat them as a useful diagnostic, not a definitive account of the model's reasoning.
Code Example
A self-contained NumPy implementation of scaled dot-product self-attention on the four-token example above. It builds Q, K and V by projecting random token embeddings, applies softmax(QKᵀ/√d_k)V, and prints the real attention matrix.
import numpy as np
np.random.seed(0)
# 4 tokens, each already embedded as a 6-dimensional vector.
tokens = ["The", "animal", "crossed", "road"]
n, d_model, d_k = len(tokens), 6, 4
X = np.random.randn(n, d_model)
# Learned projections that turn each token into a Query, Key and Value.
W_q = np.random.randn(d_model, d_k)
W_k = np.random.randn(d_model, d_k)
W_v = np.random.randn(d_model, d_k)
Q, K, V = X @ W_q, X @ W_k, X @ W_v
# Scaled dot-product attention: softmax(Q Kᵀ / √d_k) V
scores = Q @ K.T / np.sqrt(d_k) # (n, n) raw compatibility
scores = scores - scores.max(axis=1, keepdims=True) # numerical stability
weights = np.exp(scores)
weights = weights / weights.sum(axis=1, keepdims=True)
output = weights @ V # (n, d_k) new representation
np.set_printoptions(precision=2, suppress=True)
print("attention matrix is", weights.shape[0], "x", weights.shape[1],
"=", weights.size, "weights")
print(weights)
print("row sums:", weights.sum(axis=1))
Running it prints a 4×4 matrix whose every row sums to 1 — the n² = 16 weights for this four-token sequence:
attention matrix is 4 x 4 = 16 weights
[[0.32 0. 0.62 0.06]
[0.88 0. 0.11 0.01]
[0.94 0. 0.06 0. ]
[0. 0.06 0.13 0.81]]
row sums: [1. 1. 1. 1.]
Swap the constant n = 4 for n = 1000 and the printed matrix would carry 1,000,000 numbers instead of 16 — the same code, the same formula, and the quadratic cost made concrete.