Embedding

Definition

An embedding is a list of numbers — a few hundred to a few thousand floating-point values — that a trained model assigns to a piece of text, an image or an item, arranged so that the geometric distance between two lists reflects how related the two things are. That last clause is the entire trick, and it is the only property separating an embedding from any other numeric encoding: a hash of "dog" and a hash of "puppy" are also just numbers, but they land in unrelated places and nothing about their arrangement can be measured. In an embedding space they land near each other, and "near" is something you can compute.

The numbers themselves mean nothing individually. Coordinate 412 does not stand for "formality" or "animal-ness"; the axes are whatever the training run happened to settle on, and they differ between two runs of the same model. Only comparisons within one model's output space carry information. This has a hard practical consequence people discover late: you cannot compare a vector from one model against a vector from another, and you cannot compare vectors produced before and after a model upgrade. Changing embedding models means recomputing every vector you have stored — which for a corpus of ten million chunks is a re-embedding bill and a full index rebuild, not a configuration change.

How It Works

Where the geometry comes from

Nothing tells the model that "dog" and "puppy" are related. The arrangement falls out of a training objective that makes proximity the cheapest way to succeed.

The first generation of text embeddings exploited the distributional hypothesis — words appearing in the same contexts tend to mean similar things. Word2Vec (Mikolov et al., 2013) trains a network to predict a word's neighbours from the word, or the word from its neighbours; GloVe (Pennington et al., 2014) factorises the global co-occurrence matrix directly. Either way, two words that keep the same company are pushed into the same region, because one shared position solves both of their prediction problems at once.

Modern text embedding models are trained contrastively instead. You assemble pairs that should be close — a question and the passage answering it, a sentence and its paraphrase, a caption and its image — and train so each pair's similarity beats that query's similarity against every other item in the batch. With a batch of 1,024 pairs, each example arrives with one positive and 1,023 negatives for free, which is why contrastive training scales with batch size rather than with hand-labelled data.

The consequence for anyone building retrieval: "similar" means exactly what the training pairs said it meant. A model trained on question–answer pairs places a question near its answer, two texts that share almost no vocabulary. A model trained on sentence paraphrases places a question near other questions. Swap one for the other and your retriever silently starts returning a different kind of thing, with no error and no drop in any metric you were watching.

Why cosine similarity, and not distance

Take a two-dimensional toy space where the first axis is "about pets" and the second is "about software", and three documents in it:

The long article says the same thing as the short note, at ten times the length, so it points in the same direction with ten times the magnitude. Now measure straight-line distance:

• short note ↔ long article: √(27² + 36²) = √2025 = 45.0
• short note ↔ database note: √(7² + 1²) = √50 = 7.07

Euclidean distance says the dog note is 6.4× closer to the database note than to the other document about dogs. The ranking is exactly backwards, and length is the only reason.

Cosine similarity divides the dot product by both lengths, which removes magnitude entirely:

• cos(short, long) = (3·30 + 4·40) / (5 · 50) = 250 / 250 = 1.00
• cos(short, database) = (3·−4 + 4·3) / (5 · 5) = 0 / 25 = 0.00

Identical direction scores 1; perpendicular scores 0. That is the answer people are given, but it is not really a choice between two metrics. Normalise every vector to length 1 first and the distinction collapses:

‖a − b‖² = ‖a‖² + ‖b‖² − 2(a·b) = 1 + 1 − 2 cos(a, b) = 2 − 2 cos(a, b)

Squared distance is now a strictly decreasing function of cosine, so the two produce the same ranking, and the dot product alone is the cosine. This is why production pipelines call something like FAISS's normalize_L2 and then use a plain inner-product index — they get cosine ranking at the cost of a multiply. OpenAI's API returns vectors already normalised to unit length for the same reason. The working rule is simple: normalise once at write time and the metric question stops existing; skip it and your relevance ranking quietly tracks document length.

What dimension actually buys

Three different things scale with the dimension count D, and confusing them is how teams end up paying for 3,072 dimensions they cannot use.

Storage scales linearly and bites immediately. Ten million chunks at 1,536 dimensions in float32 is 10,000,000 × 1,536 × 4 bytes = 61.4 GB — for vectors alone, before the index structure, and typically expected to sit in RAM. Per chunk that is 6,144 bytes of vector for a passage whose text may be under 2 KB: you are storing about three times more geometry than prose. Quantising each coordinate to int8 cuts it to 15.4 GB, and to a single bit per coordinate — binary embeddings — to 1.92 GB, a 32× reduction usually paired with re-ranking the top few hundred hits at full precision to recover what the rounding lost. The complementary lever, cutting D itself, belongs to dimensionality reduction.

Quality scales far more weakly than the defaults imply. Matryoshka Representation Learning (Kusupati et al., 2022) trains a vector so that every prefix of it is itself a usable embedding, making shrinkage a slice rather than a projection. OpenAI reported in January 2024 that text-embedding-3-large scores 64.6 on the MTEB average at its full 3,072 dimensions and 62.0 when truncated to 256 — still ahead of the older text-embedding-ada-002, which scores 61.0 at 1,536. A vector twelve times smaller from the better model beat a vector six times larger from the older one. Those are 2024 numbers on a leaderboard that has turned over repeatedly since, so treat the ranking as stale and the shape of the result as the durable part: dimension is not what makes an embedding good, the training objective is.

Capacity has a hard ceiling that no amount of training removes. Weller et al. (2025) connect embedding dimension to a result from learning theory: D bounds how many distinct top-k document sets a single-vector retriever can ever return. Optimising vectors freely, with no language model in the way and therefore in the best case possible, the largest corpus for which every top-2 pair is reachable by some query is 10 documents at D = 4, 28 at D = 8, 79 at D = 16 and 296 at D = 32; extrapolating their fit gives roughly 500,000 at D = 512 and 4 million at D = 1,024. On LIMIT, the 50,000-document, 1,000-query dataset they built to stress this, the best single-vector model reached 18.9% recall@100 while BM25 keyword search reached 97.8%. That gap is not a training failure. It is the shape of the container.

Types

Two distinctions in the embedding literature are real, load-bearing, and worth keeping straight.

Static versus contextual. Word2Vec and GloVe assign one vector per word type, permanently: "bank" gets a single vector that averages the river and the money together, and no amount of surrounding text changes it. Transformer encoders such as BERT produce a vector per occurrence, so "bank" in a sentence about loans and "bank" in a sentence about rivers land in different places. Every current embedding API is contextual underneath — but it pools those per-token vectors into a single vector per input, which means a 500-token chunk is compressed into one point whether it makes one argument or seven. That pooling step, not the model, is the reason chunk size matters so much in RAG: a chunk covering several topics ends up at their average, near none of them.

Single-vector versus multi-vector. Late-interaction models in the ColBERT family keep one vector per token and score a document by summing, over query tokens, the best match among the document's tokens. It costs far more storage and a more complicated index, and it is the direct structural answer to the capacity ceiling above — on the same LIMIT benchmark where the best single-vector model reached 18.9% recall@100, a ColBERT-style model reached 54.8%.

Real-World Applications

Retrieval-augmented generation. A corpus is split into chunks, each embedded and stored in a vector search index; at query time the question is embedded with the same model and the nearest chunks are pasted into the prompt. Nearly every "chat with your documents" product is this loop, and the embedding model determines what it can find — which is why production systems increasingly run dense retrieval alongside BM25 and fuse the results, hedging exactly the failure the LIMIT numbers expose.

Candidate generation in recommenders. YouTube's deep recommendation system (Covington et al., 2016) learns an embedding for each video and each user session and reduces "which of millions of videos should we consider" to an approximate nearest-neighbour lookup in that space, with a heavier ranking model applied only to the few hundred survivors. The two-stage structure — cheap embedding retrieval, expensive reranking — is now standard across recommendation systems.

Cross-modal search and zero-shot classification. CLIP (Radford et al., 2021) trains an image encoder and a text encoder contrastively on 400 million image–caption pairs so both produce vectors in one shared space. That single property converts image classification into a nearest-neighbour problem: embed the sentence "a photo of a tabby cat" and compare it to the image vector, with no training on cats required. It is also what makes text-to-image search work in computer vision pipelines, and the same recipe now underlies audio and video embedding models.

Key Concepts

• A cosine score has no absolute meaning. 0.82 is not "82% similar", and it is not comparable across models — calibrate any threshold against random pairs from your own corpus.
• Query and document may need different treatment. Asymmetric models such as the E5 and BGE families expect literal query: and passage: prefixes; getting them backwards costs recall silently, with no error anywhere.
• Pooling is the model's decision, not yours. Mean-pooling a model trained for CLS pooling, or the reverse, degrades the space. Use whatever the model card specifies.
• Hybrid retrieval is the default, not a fallback. Dense embeddings miss exact identifiers, rare names and negation; BM25 misses paraphrase. Fusing both rankings covers more than either.

Challenges

The space is a narrow cone, not a ball. Ethayarajh (2019) measured the average cosine similarity between uniformly randomly sampled words and found it far above zero in every layer of ELMo, BERT and GPT-2 — roughly 0.6 through GPT-2's middle layers, rising toward almost perfect similarity by the last, where "any two words have on average an almost perfect cosine similarity". Vectors are not spread evenly over the sphere; they crowd into a thin region of it. So the intuition that 0.7 must indicate a strong match is unfounded until you know what two unrelated texts score in that particular model. Measure the baseline and read every score relative to it, or centre and whiten the vectors before comparing. (This is a distinct problem from the distance concentration described under dimensionality reduction, which is about uniformly scattered points; here the points are anything but uniform.)

The famous analogy is partly an artefact of the evaluation code. "king − man + woman ≈ queen" is the demonstration everyone remembers, and the standard procedure quietly excludes the three input words from the candidate list before picking a nearest neighbour. Remove that exclusion and the vector closest to king − man + woman is king — a point made by Linzen (2016) and again by Nissim et al. (2020), who argue on that basis that analogy tests are a poor instrument for measuring bias. Embeddings do encode a great deal of relational structure. They encode less of it than the party trick implies.

Nearest-neighbour retrieval cannot abstain, and negation barely registers. Similarity search always returns a top-k; there is no "nothing here matches", so a question your corpus cannot answer still yields confident-looking context for the model to build on. Worse, "the drug is safe for children" and "the drug is not safe for children" share nearly every token and are separated by one word that the contrastive training objective was rarely forced to notice, so they sit close together. Both problems are why a cross-encoder reranker over the top 50–100 hits is standard in serious systems: it reads the query and passage together instead of comparing two vectors that were computed in ignorance of each other.

Bias is stored as geometry and travels downstream intact. Caliskan et al. (2017) showed in Science that word embeddings trained on web text reproduce documented human implicit-association effects — female names sitting closer to family terms and male names closer to career terms — at effect sizes matching the human psychology literature. Because an embedding is a substrate rather than an endpoint, that structure is inherited by every classifier, ranker and retrieval system built on top, which is a specific mechanism behind algorithmic bias rather than a general worry about training data.

Future Trends

Dimension is becoming a request parameter rather than a model property: Matryoshka training is increasingly baked in and dimensions is now an argument on major embedding APIs, moving the choice from procurement time to query time. Quantisation is heading the same way, with models trained to survive int8 or binary storage rather than being rounded after the fact.

Multi-vector and late-interaction retrieval is moving out of research and into mainstream vector databases, driven by exactly the capacity ceiling measured above — if one vector per document cannot express every useful top-k set, more vectors per document is the structural fix, not a bigger D. And instruction-conditioned models, which produce a different space depending on a task description supplied with the input, are turning "which embedding model do I choose" into "what did I ask it to encode" — changeable without re-selecting a model, though never without re-embedding the corpus.

Code Example

The worked example above, in NumPy. The point is the last four lines: once the vectors are normalised, cosine and Euclidean distance stop being alternatives.

import numpy as np

short_dog = np.array([3.0, 4.0])     # a short note about dogs
long_dog  = np.array([30.0, 40.0])   # the same content, ten times as long
database  = np.array([-4.0, 3.0])    # a note about databases

euclid = lambda a, b: np.linalg.norm(a - b)
cosine = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(euclid(short_dog, long_dog))    # 45.0   <- "far"
print(euclid(short_dog, database))    #  7.07  <- "near", and wrong
print(cosine(short_dog, long_dog))    #  1.0
print(cosine(short_dog, database))    #  0.0

# Normalise once, and the choice of metric disappears:
unit = lambda v: v / np.linalg.norm(v)
a, b, c = unit(short_dog), unit(long_dog), unit(database)

print(euclid(a, b) ** 2, 2 - 2 * cosine(a, b))   # 0.0 0.0
print(euclid(a, c) ** 2, 2 - 2 * cosine(a, c))   # 2.0 2.0

Squared distance between unit vectors equals 2 − 2·cosine, exactly, so any ranking by one is a ranking by the other. This is the identity that lets a vector index store normalised vectors and compute a bare dot product while honestly reporting cosine similarity.