Definition
A Vision Transformer (ViT) is the transformer architecture built for language, applied to images with almost none of the machinery that vision models had relied on for a decade. Instead of sliding convolutional filters over pixels, a ViT cuts the image into a grid of fixed-size square patches, treats each patch as if it were a word, and feeds the sequence through a standard transformer encoder — the same stack of self-attention and feed-forward layers used in a language model.
The counter-intuitive part is that this works at all. A convolutional network has spatial reasoning built into its wiring; a transformer, applied this way, starts out knowing nothing about images — not that pixels near each other are related, not that an object means the same thing wherever it appears. It has to learn all of that from data. The bet the architecture makes, and the reason it matters, is that given enough data this ignorance becomes an advantage: a model that assumes less can be corrected more, and it keeps improving as you pour in scale long after a convolutional network's built-in assumptions have become a ceiling. That single trade — weaker built-in assumptions for better scaling — is the whole story, and everything below is a consequence of it.
How It Works
The pipeline has four steps, and the first three exist only to turn a two-dimensional image into the flat sequence of tokens a transformer expects.
Patching. The image is divided into a regular grid of non-overlapping squares of side P pixels — commonly 16. A 224x224 image at P = 16 becomes a 14x14 grid of 196 patches. Nothing overlaps and nothing is learned yet; this is pure reshaping.
Linear embedding. Each patch — for a 16x16 colour patch that is 16 x 16 x 3 = 768 raw numbers — is flattened into a vector and passed through a single learned linear projection to a fixed model width D. The result is one embedding vector per patch, exactly analogous to the token embeddings a language model looks up for words. A small transformer might use D = 768; the projection is shared across all patches.
Position information. Flattening threw away where each patch sat in the image, and self-attention is order-agnostic — it would treat a shuffled sequence identically. So a learned position embedding is added to each patch vector, one per grid slot, restoring the "this patch was top-left" signal. The original ViT used plain learnable one-dimensional position embeddings and found that fancier two-dimensional variants bought almost nothing; the broader idea of injecting order into an order-blind model is positional encoding. A single extra learnable vector, the classification token, is usually prepended to the sequence; its output state, after passing through the whole encoder, becomes the image's summary representation for a classifier.
The encoder. The resulting sequence — 197 vectors for our example, the 196 patches plus the class token — runs through a stack of identical transformer blocks. Each block applies self-attention, in which every token computes a weighted combination of every other token, followed by a small feed-forward network, with layer normalization before each and a residual connection around each. This is where the model actually learns that certain patches belong together — an edge continuing across a patch boundary, two halves of a face — because attention lets any patch pull information from any other, regardless of distance. A convolution can only ever look at its immediate neighbourhood; a ViT sees the whole image from layer one.
The token count drives everything
The number of tokens is not a detail — it sets the model's entire compute cost, and it follows directly from the patch size. For an image of height H and width W cut into patches of side P:
number of tokens N = (H / P) x (W / P) = HW / P²Self-attention compares every token with every other, so a single attention layer builds an N x N table of scores. For our 196 patches that is 196² = 38,416 pairwise comparisons per attention head per layer, and the compute grows with N². Substituting the formula above, attention cost scales as (HW / P²)² — which means it grows with the fourth power of 1/P. Halve the patch size and you quadruple the token count and multiply attention cost sixteen-fold; the same sixteen-fold jump comes from doubling the image resolution at a fixed patch size, because both change N by the same factor of four.
This is why patch size is the single most consequential knob on a ViT. A smaller patch sees finer detail but explodes the sequence length; a larger patch is cheap but coarse. It is also why naively running a transformer over raw pixels — one token per pixel — was never viable: a 224x224 image has 50,176 pixels, and 50,176² is over two billion pairwise scores per layer. Patching is the trick that makes the sequence short enough to afford.
Real-World Applications
The ViT's most consequential deployment is not standalone image classification but its role as the vision encoder inside multimodal systems. OpenAI's CLIP, released in 2021, pairs a text encoder with a ViT image encoder (variants such as ViT-B/32 and ViT-L/14) trained to place matching image-text pairs near each other in a shared space; that CLIP image encoder became the eyes of a generation of systems, feeding the image conditioning in text-to-image generation pipelines and serving as the visual front-end that many multimodal AI language models bolt onto an LLM to let it "see". When a chat assistant describes a photo you upload, a ViT-style encoder is very often the component turning that photo into tokens the language model can read.
It also anchors general-purpose vision backbones. Meta's Segment Anything Model (SAM, 2023) uses a large ViT image encoder — pre-trained with masked-autoencoder self-supervision — to produce a single image embedding that its lightweight decoder queries to segment any object on demand. Self-supervised ViT features from the DINO and DINOv2 line are used off the shelf for retrieval, depth estimation and dense correspondence without task-specific training. The common thread is that a ViT pre-trained on enough data yields a general representation that many downstream tasks reuse — the same "pre-train once, transfer everywhere" pattern that made transformers dominant in language.
Key Concepts
Inductive bias is the thing being traded. Inductive bias is what a model assumes before seeing any data. A CNN's is strong and specific — as the ViT paper puts it, "locality, two-dimensional neighborhood structure, and translation equivariance are baked into each layer throughout the whole model." A ViT has "much less image-specific inductive bias": only its feed-forward layers are local, its attention is global from the start, and "all spatial relations between the patches have to be learned from scratch." A weak prior is a liability when data is scarce and an asset when it is abundant, because a strong prior that is slightly wrong eventually becomes the thing holding a model back.
The data threshold is where the two curves cross. The original 2020 paper measured this directly. Trained on ImageNet-1k alone — 1.3 million images — Vision Transformers landed "a few percentage points below ResNets of comparable size". Pre-trained instead on the in-house JFT-300M dataset (303 million images across 18k classes) and then fine-tuned, the same architectures overtook state-of-the-art convolutional networks; the paper's summary is blunt — "large scale training trumps inductive bias" — and it puts the regime where the picture changes at roughly "14M-300M images". Below that, the CNN's built-in assumptions are worth more than anything the ViT can learn; above it, they are a constraint the ViT does not carry.
Scaling is the payoff. Because it assumes so little, a ViT keeps rewarding more data, more parameters and more compute. In the 2020 paper the best model, ViT-H/14 pre-trained on JFT-300M, reached 88.55% top-1 accuracy on ImageNet, and did so using substantially less pre-training compute than the convolutional baselines it beat — the ViT-L/16 variant matched Big Transfer's large ResNet while spending roughly 0.68k TPUv3-core-days against its 9.9k. The variants scale cleanly: ViT-Base has about 86 million parameters, ViT-Large about 307 million, and ViT-Huge about 632 million (figures from the original paper). The notation ViT-L/16 reads as "the Large variant with 16-pixel patches"; because sequence length is inversely proportional to the square of the patch size, ViT-L/16 and the finer-grained ViT-L/14 are the same model at different compute budgets.
Where each is preferred today. The honest 2026 summary is neither "CNNs are obsolete" nor "CNNs still win". Convolutional networks remain the sensible default when labelled data is scarce — medical imaging, where a dataset is hundreds or a few thousand studies because each costs a radiologist's time — or when the compute budget is tight, as in real-time detection on embedded hardware. The pure-attention ViT wins when large-scale pre-training is available and the representation will be transferred widely, which is exactly the multimodal setting. And the operator gap turned out to be smaller than the hype: a modern convolutional recipe like ConvNeXt, which borrows the training tricks that made transformers shine, matches a comparable transformer on mid-sized data using nothing but convolutions. Most systems that actually ship are hybrids — a convolutional stem for the cheap high-resolution early stages and attention only once the feature map has shrunk, or windowed-attention designs that reintroduce locality by hand — rather than a purist choice of one architecture.
Challenges
- Data hunger is structural, not incidental. A ViT trained from scratch on a mid-sized labelled dataset will usually underperform a CNN of the same size, and no amount of regularization fully closes the gap — the fix is more data or strong pre-training, which is not always available. Reaching for a from-scratch ViT on a few thousand images is the most common way to be disappointed by one.
- Quadratic cost caps resolution. Because attention scales with the square of the token count and the token count grows with resolution, a pure global-attention ViT becomes expensive fast on large images. High-resolution dense tasks — segmentation, detection — are where the
N²term bites hardest, and it is the reason windowed and hierarchical variants exist at all. - Patch boundaries are an arbitrary grid. Cutting the image into a fixed grid can slice an object across several patches with no regard for its shape, and the model must stitch it back together through attention. Fine structure smaller than a patch is only ever seen pre-mixed inside a single token, which is one reason very small patch sizes help on detail-heavy tasks despite their cost.
- Interpretability shifts, it does not vanish. A CNN's early filters have a legible meaning — edges, textures — that a ViT's global attention does not offer in the same way. Attention maps are suggestive but are not a faithful account of what drove a prediction, so debugging a ViT's mistakes is its own skill rather than a transfer of CNN intuition.
Code Example
The relationship between patch size, resolution and attention cost is worth seeing on real numbers. This computes the token count N = HW/P² for several configurations and the resulting attention cost relative to the standard 224x224 / patch-16 setup, whose cost scales as N²:
def tokens(res, p):
return (res // p) ** 2
configs = [(224, 16), (224, 8), (384, 16), (448, 16), (224, 14)]
base = tokens(224, 16)
for res, p in configs:
n = tokens(res, p)
print(f"{res}x{res}, patch {p:>2}: {n:>4} tokens, "
f"attention cost {(n / base) ** 2:>5.1f}x baseline")
Output:
224x224, patch 16: 196 tokens, attention cost 1.0x baseline
224x224, patch 8: 784 tokens, attention cost 16.0x baseline
384x384, patch 16: 576 tokens, attention cost 8.6x baseline
448x448, patch 16: 784 tokens, attention cost 16.0x baseline
224x224, patch 14: 256 tokens, attention cost 1.7x baseline
The last two rows make the 1/P⁴ scaling concrete: halving the patch from 16 to 8 and doubling the resolution from 224 to 448 both land on 784 tokens and the same sixteen-fold attention cost, because each changes N by a factor of four and attention pays the square of that. This is the arithmetic every ViT design decision is negotiating against.