Definition
A transformer is a neural-network architecture, introduced in 2017, that processes an entire sequence in parallel using self-attention instead of stepping through it one element at a time. This is the change that mattered: earlier sequence models such as recurrent neural networks read a sentence left to right, so position 100 could not be computed until position 99 was done, and a dependency between the first and last word had to survive dozens of sequential steps. A transformer connects any two positions directly, in a single operation, and does the whole sequence at once — and it is that parallelism, not just the accuracy, that let training scale onto GPUs and made today's large language models possible.
It came from the paper "Attention Is All You Need" (Vaswani et al., 2017), whose title is the whole argument: the authors dropped recurrence and convolution entirely and kept only attention. One clarification before the mechanism, because search traffic conflates them: the transformer is the architecture described here, while Hugging Face transformers is a separate Python library that packages implementations of many transformer models. If you pip install transformers, you are installing that library; this page is about the architecture it implements.
How It Works
A transformer turns a sequence of tokens into a sequence of vectors and refines those vectors, layer after identical layer, until the final representation is good enough to predict the next token or classify the input. Five ideas do the work.
Token and positional embeddings — why position has to be injected
Each token is first mapped to a vector by an embedding table, the same way most language models begin. But there is a catch that is specific to this architecture: self-attention is order-blind. The attention operation is a weighted sum over the other tokens, and a sum does not care about order — shuffle the input words and, without extra information, the output is the same set of vectors in a different arrangement. A recurrent network never had this problem because it reads in order; a transformer sees everything at once and so has no inherent sense of first, second or last.
The fix is to add position information directly into the embeddings before the first layer. The original paper uses fixed sinusoidal positional encodings: for position pos and dimension i, PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) and the odd dimensions use cosine. Each position gets a distinct pattern of sine and cosine waves at different frequencies, so the model can tell positions apart and, because the functions are smooth, generalise to relative distances. Many later models swap this for learned or rotary position embeddings, but the principle is invariant: if you do not inject position, attention cannot use it.
Multi-head self-attention
The core layer is self-attention: each token forms a Query, and every token offers a Key and a Value; a token's Query is compared against all the Keys, the scores are normalised with a softmax, and the output is the relevance-weighted blend of the Values. The whole sequence is computed at once as Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V — see the self-attention page for the derivation and a worked matrix, which this page does not repeat.
"Multi-head" means the model does not run this once but several times in parallel, each head using its own learned projection of the input. One head can specialise in tracking subject-verb agreement while another follows what a pronoun refers to; the heads' outputs are concatenated and mixed back together. In the base model there are h = 8 heads, each operating on a d_k = 64-dimensional slice of the d_model = 512 representation — 8 × 64 = 512, so the heads partition the model width rather than adding to it.
Feed-forward layers
After attention has mixed information between tokens, each token is passed, independently, through a small two-layer fully-connected network — a position-wise feed-forward network. It expands the representation to a wider inner dimension, applies a non-linearity, and projects back. In the base model that inner width is d_ff = 2048, four times d_model. This is where a large share of a transformer's parameters and compute actually live: because d_ff = 4 · d_model, the feed-forward block holds roughly twice as many weights as the attention block in the same layer (the Code Example works this out).
Residual connections and layer normalisation
Every attention block and every feed-forward block is wrapped in a residual connection — the block's input is added back to its output — followed by layer normalisation. The residual path lets gradients flow straight through a deep stack without vanishing, and the normalisation keeps activations at a stable scale from layer to layer. Without both, stacking many layers would not train.
Stacking identical blocks — and what it costs
A transformer is these blocks repeated. The base model stacks N = 6 of them in the encoder and 6 in the decoder; scaling the architecture up mostly means making d_model, d_ff and N larger. The load-bearing cost sits in the attention step: comparing every token with every other token is O(n²) in the sequence length. Ten times the input is a hundred times the attention compute and, naively, a hundred times the memory for the score matrix (the self-attention page shows the n² count concretely). This single fact is why a longer context window is expensive, and why an entire research industry — FlashAttention, sparse and sliding-window attention, linear attention — exists to soften that quadratic.
Types
Transformers come in three genuine architectural families, distinguished by which halves of the original encoder-decoder design they keep. This is a real, widely-used taxonomy, not a stylistic label.
- Encoder-only (BERT). Keeps only the encoder, with unmasked attention so every token sees the whole input in both directions at once. Good for understanding — classification, retrieval, named-entity recognition, sentence embeddings — but it does not generate text left to right.
- Decoder-only (GPT-style LLMs). Keeps only the decoder, with causal (masked) attention so each token can attend only to earlier tokens. That mask is exactly what makes autoregressive next-token generation valid, which is why nearly all modern chat and completion LLMs are decoder-only.
- Encoder-decoder (T5, and the original 2017 Transformer). An encoder reads the input and a decoder generates the output while cross-attending back into the encoder's representation. This is the natural shape for turning one sequence into another — translation, summarization — and it is the configuration the original paper was built and benchmarked in.
Real-World Applications
The original 2017 model was a translation system: the base and big transformers were trained on the WMT 2014 English-German corpus of about 4.5 million sentence pairs, and the big model reached 28.4 BLEU on English-German and a then-state-of-the-art 41.8 BLEU on English-French, after 3.5 days of training on eight NVIDIA P100 GPUs. Translation is still a flagship use — Google Translate moved to transformer-based systems — but the architecture spread far past it:
- Large language models. GPT, Claude, Gemini, Llama and essentially every current LLM are decoder-only transformers; the architecture is what "the model" refers to when people discuss them. They power chat assistants, coding tools like GitHub Copilot, and content generation.
- Bidirectional understanding. BERT and its descendants (encoder-only transformers) sit behind search ranking, text classification, and the sentence-embedding models used for semantic search and retrieval-augmented pipelines.
- Vision. The Vision Transformer (ViT) cuts an image into fixed-size patches and treats them as a sequence of tokens, bringing the same self-attention to image classification and object detection.
- Science. AlphaFold uses attention-based modules over sequences of amino acids as part of predicting protein folding structures — an application far from the natural-language task the architecture was invented for.
Challenges
The quadratic cost is the defining limitation. Because attention is O(n²), the price of long context is real and non-linear: at 1,000 tokens the attention matrix already holds a million scores, and doubling the input more than doubles the cost of using it. Every practical long-context system is, in part, an answer to this — from FlashAttention (which keeps the exact result but never writes the full matrix to slow memory) to sparse and windowed patterns (which drop some token-pairs to buy near-linear scaling).
Position generalisation is fragile. Because position is injected rather than intrinsic, a model trained at one context length does not automatically work at a longer one — the position signals it sees at inference can fall outside the range it was trained on. Extending context length is an active engineering problem, not a free parameter.
Compute and data intensity. The parallelism that makes transformers trainable also makes them hungry: the architecture rewards scale, which means large GPU clusters, large datasets, and correspondingly large energy costs to both train and serve frontier models.
Attention is not an explanation. It is tempting to read attention weights as "what the model looked at," but a large weight is not proof that a token caused the output, and attention maps can be altered without changing predictions. They are a useful diagnostic, not a definitive account of the model's reasoning.
Code Example
A common question is where a transformer's parameters actually go. This snippet counts the weights in one base-model layer (d_model = 512, d_ff = 2048) and shows the durable relationship: with the feed-forward inner width set to four times the model width, the feed-forward block holds twice as many parameters as attention.
d_model = 512 # base transformer model width
d_ff = 2048 # feed-forward inner width (4 * d_model)
# Self-attention: four square projections (Q, K, V, and the output mix),
# each d_model x d_model. Biases are negligible and omitted here.
attention_params = 4 * d_model * d_model
# Position-wise feed-forward: two layers, d_model->d_ff then d_ff->d_model.
ffn_params = 2 * d_model * d_ff
print("attention params:", attention_params) # 1,048,576
print("feed-forward params:", ffn_params) # 2,097,152
print("ffn / attention ratio:", ffn_params / attention_params) # 2.0
The ratio comes out to exactly 2.0 whenever d_ff = 4 · d_model, which is the standard choice: 2 · d_model · (4 · d_model) = 8 · d_model² for the feed-forward block versus 4 · d_model² for attention. It is a small piece of arithmetic, but it explains a fact that surprises people — in a plain transformer, more parameters sit in the "simple" feed-forward layers than in the attention everyone talks about.