---
source: 'https://howaiworks.ai/glossary/embedding'
section: glossary
title: Embedding
description: >-
  What an embedding vector actually is, why cosine similarity beats Euclidean
  distance, what dimensions buy you, and where the geometry misleads you.
tags:
  - embedding
  - vector search
  - representation learning
  - semantic search
  - NLP
  - Machine Learning
category: Machine Learning
datePublished: '2025-07-16'
lastUpdated: '2026-07-22'
---

# Embedding

> What an embedding vector actually is, why cosine similarity beats Euclidean distance, what dimensions buy you, and where the geometry misleads you.

## Definition

An embedding is a list of numbers — a few hundred to a few thousand floating-point values — that a
trained model assigns to a piece of text, an image or an item, arranged so that **the geometric
distance between two lists reflects how related the two things are**. That last clause is the
entire trick, and it is the only property separating an embedding from any other numeric encoding:
a hash of "dog" and a hash of "puppy" are also just numbers, but they land in unrelated places and
nothing about their arrangement can be measured. In an embedding space they land near each other,
and "near" is something you can compute.

The numbers themselves mean nothing individually. Coordinate 412 does not stand for "formality" or
"animal-ness"; the axes are whatever the training run happened to settle on, and they differ
between two runs of the same model. Only *comparisons within one model's output space* carry
information. This has a hard practical consequence people discover late: you cannot compare a
vector from one model against a vector from another, and you cannot compare vectors produced
before and after a model upgrade. Changing embedding models means recomputing every vector you
have stored — which for a corpus of ten million chunks is a re-embedding bill and a full index
rebuild, not a configuration change.

## How It Works

### Where the geometry comes from

Nothing tells the model that "dog" and "puppy" are related. The arrangement falls out of a training
objective that makes proximity the cheapest way to succeed.

The first generation of text embeddings exploited the distributional hypothesis — words appearing in
the same contexts tend to mean similar things. Word2Vec ([Mikolov et al.,
2013](https://arxiv.org/abs/1301.3781)) trains a network to predict a word's neighbours from the
word, or the word from its neighbours; GloVe ([Pennington et al.,
2014](https://aclanthology.org/D14-1162/)) factorises the global co-occurrence matrix directly.
Either way, two words that keep the same company are pushed into the same region, because one
shared position solves both of their prediction problems at once.

Modern text embedding models are trained **contrastively** instead. You assemble pairs that should
be close — a question and the passage answering it, a sentence and its paraphrase, a caption and
its image — and train so each pair's similarity beats that query's similarity against every *other*
item in the batch. With a batch of 1,024 pairs, each example arrives with one positive and 1,023
negatives for free, which is why contrastive training scales with batch size rather than with
hand-labelled data.

The consequence for anyone building retrieval: **"similar" means exactly what the training pairs
said it meant.** A model trained on question–answer pairs places a question near its *answer*, two
texts that share almost no vocabulary. A model trained on sentence paraphrases places a question
near *other questions*. Swap one for the other and your retriever silently starts returning a
different kind of thing, with no error and no drop in any metric you were watching.

### Why cosine similarity, and not distance

Take a two-dimensional toy space where the first axis is "about pets" and the second is "about
software", and three documents in it:

| document | vector | length |
|---|---|---|
| short note about dogs | [3, 4] | 5 |
| long article about dogs | [30, 40] | 50 |
| note about databases | [−4, 3] | 5 |

The long article says the same thing as the short note, at ten times the length, so it points in
the same direction with ten times the magnitude. Now measure straight-line distance:

- short note ↔ long article: √(27² + 36²) = √2025 = **45.0**
- short note ↔ database note: √(7² + 1²) = √50 = **7.07**

Euclidean distance says the dog note is **6.4× closer to the database note** than to the other
document about dogs. The ranking is exactly backwards, and length is the only reason.

Cosine similarity divides the dot product by both lengths, which removes magnitude entirely:

- cos(short, long) = (3·30 + 4·40) / (5 · 50) = 250 / 250 = **1.00**
- cos(short, database) = (3·−4 + 4·3) / (5 · 5) = 0 / 25 = **0.00**

Identical direction scores 1; perpendicular scores 0. That is the answer people are given, but it
is not really a choice between two metrics. Normalise every vector to length 1 first and the
distinction collapses:

‖a − b‖² = ‖a‖² + ‖b‖² − 2(a·b) = 1 + 1 − 2 cos(a, b) = **2 − 2 cos(a, b)**

Squared distance is now a strictly decreasing function of cosine, so the two produce the *same
ranking*, and the dot product alone *is* the cosine. This is why production pipelines call
something like FAISS's `normalize_L2` and then use a plain inner-product index — they get cosine
ranking at the cost of a multiply. OpenAI's API returns vectors already normalised to unit length
for the same reason. The working rule is simple: normalise once at write time and the metric
question stops existing; skip it and your relevance ranking quietly tracks document length.

### What dimension actually buys

Three different things scale with the dimension count *D*, and confusing them is how teams end up
paying for 3,072 dimensions they cannot use.

**Storage scales linearly and bites immediately.** Ten million chunks at 1,536 dimensions in
float32 is 10,000,000 × 1,536 × 4 bytes = **61.4 GB** — for vectors alone, before the index
structure, and typically expected to sit in RAM. Per chunk that is 6,144 bytes of vector for a
passage whose text may be under 2 KB: you are storing about three times more geometry than prose.
[Quantising](https://howaiworks.ai/glossary/quantization) each coordinate to int8 cuts it to 15.4 GB, and to a single
bit per coordinate — binary embeddings — to 1.92 GB, a 32× reduction usually paired with re-ranking
the top few hundred hits at full precision to recover what the rounding lost. The complementary
lever, cutting *D* itself, belongs to
[dimensionality reduction](https://howaiworks.ai/glossary/dimensionality-reduction).

**Quality scales far more weakly than the defaults imply.** Matryoshka Representation Learning
([Kusupati et al., 2022](https://arxiv.org/abs/2205.13147)) trains a vector so that every prefix of
it is itself a usable embedding, making shrinkage a slice rather than a projection. OpenAI reported
in January 2024 that `text-embedding-3-large` scores 64.6 on the MTEB average at its full 3,072
dimensions and **62.0 when truncated to 256** — still ahead of the older `text-embedding-ada-002`,
which scores 61.0 at 1,536. A vector twelve times smaller from the better model beat a vector six
times larger from the older one. Those are 2024 numbers on a leaderboard that has turned over
repeatedly since, so treat the ranking as stale and the shape of the result as the durable part:
dimension is not what makes an embedding good, the training objective is.

**Capacity has a hard ceiling that no amount of training removes.** [Weller et al.
(2025)](https://arxiv.org/abs/2508.21038) connect embedding dimension to a result from learning
theory: *D* bounds how many distinct top-*k* document sets a single-vector retriever can ever
return. Optimising vectors freely, with no language model in the way and therefore in the best case
possible, the largest corpus for which every top-2 pair is reachable by *some* query is 10
documents at *D* = 4, 28 at *D* = 8, 79 at *D* = 16 and 296 at *D* = 32; extrapolating their fit
gives roughly 500,000 at *D* = 512 and 4 million at *D* = 1,024. On LIMIT, the 50,000-document,
1,000-query dataset they built to stress this, the best single-vector model reached **18.9%
recall@100 while BM25 keyword search reached 97.8%.** That gap is not a training failure. It is the
shape of the container.

## Types

Two distinctions in the embedding literature are real, load-bearing, and worth keeping straight.

**Static versus contextual.** Word2Vec and GloVe assign one vector per word type, permanently:
"bank" gets a single vector that averages the river and the money together, and no amount of
surrounding text changes it. [Transformer](https://howaiworks.ai/glossary/transformer) encoders such as BERT produce a
vector per *occurrence*, so "bank" in a sentence about loans and "bank" in a sentence about rivers
land in different places. Every current embedding API is contextual underneath — but it pools those
per-token vectors into a single vector per input, which means a 500-token chunk is compressed into
one point whether it makes one argument or seven. That pooling step, not the model, is the reason
chunk size matters so much in [RAG](https://howaiworks.ai/glossary/retrieval-augmented-generation): a chunk covering
several topics ends up at their average, near none of them.

**Single-vector versus multi-vector.** Late-interaction models in the ColBERT family keep one
vector per token and score a document by summing, over query tokens, the best match among the
document's tokens. It costs far more storage and a more complicated index, and it is the direct
structural answer to the capacity ceiling above — on the same LIMIT benchmark where the best
single-vector model reached 18.9% recall@100, a ColBERT-style model reached 54.8%.

## Real-World Applications

**Retrieval-augmented generation.** A corpus is split into chunks, each embedded and stored in a
[vector search](https://howaiworks.ai/glossary/vector-search) index; at query time the question is embedded with the
same model and the nearest chunks are pasted into the prompt. Nearly every "chat with your
documents" product is this loop, and the embedding model determines what it can find — which is why
production systems increasingly run dense retrieval alongside BM25 and fuse the results, hedging
exactly the failure the LIMIT numbers expose.

**Candidate generation in recommenders.** YouTube's deep recommendation system ([Covington et al.,
2016](https://research.google/pubs/deep-neural-networks-for-youtube-recommendations/)) learns an
embedding for each video and each user session and reduces "which of millions of videos should we
consider" to an approximate nearest-neighbour lookup in that space, with a heavier ranking model
applied only to the few hundred survivors. The two-stage structure — cheap embedding retrieval,
expensive reranking — is now standard across [recommendation
systems](https://howaiworks.ai/glossary/recommendation-systems).

**Cross-modal search and zero-shot classification.** CLIP ([Radford et al.,
2021](https://arxiv.org/abs/2103.00020)) trains an image encoder and a text encoder contrastively
on 400 million image–caption pairs so both produce vectors in *one shared space*. That single
property converts image classification into a nearest-neighbour problem: embed the sentence "a
photo of a tabby cat" and compare it to the image vector, with no training on cats required. It is
also what makes text-to-image search work in [computer vision](https://howaiworks.ai/glossary/computer-vision)
pipelines, and the same recipe now underlies audio and video embedding models.

## Key Concepts

- **A cosine score has no absolute meaning.** 0.82 is not "82% similar", and it is not comparable
  across models — calibrate any threshold against random pairs from your own corpus.
- **Query and document may need different treatment.** Asymmetric models such as the E5 and BGE
  families expect literal `query:` and `passage:` prefixes; getting them backwards costs recall
  silently, with no error anywhere.
- **Pooling is the model's decision, not yours.** Mean-pooling a model trained for CLS pooling, or
  the reverse, degrades the space. Use whatever the model card specifies.
- **Hybrid retrieval is the default, not a fallback.** Dense embeddings miss exact identifiers,
  rare names and negation; BM25 misses paraphrase. Fusing both rankings covers more than either.

## Challenges

**The space is a narrow cone, not a ball.** [Ethayarajh
(2019)](https://aclanthology.org/D19-1006/) measured the average cosine similarity between
uniformly randomly sampled words and found it far above zero in every layer of ELMo, BERT and
GPT-2 — roughly 0.6 through GPT-2's middle layers, rising toward almost perfect similarity by the
last, where "any two words have on average an almost perfect cosine similarity". Vectors are not
spread evenly over the sphere; they crowd into a thin region of it. So the intuition that 0.7 must
indicate a strong match is unfounded until you know what two *unrelated* texts score in that
particular model. Measure the baseline and read every score relative to it, or centre and whiten
the vectors before comparing. (This is a distinct problem from the distance concentration described
under [dimensionality reduction](https://howaiworks.ai/glossary/dimensionality-reduction), which is about uniformly
scattered points; here the points are anything but uniform.)

**The famous analogy is partly an artefact of the evaluation code.** "king − man + woman ≈ queen"
is the demonstration everyone remembers, and the standard procedure quietly excludes the three
input words from the candidate list before picking a nearest neighbour. Remove that exclusion and
the vector closest to king − man + woman is *king* — a point made by [Linzen
(2016)](https://aclanthology.org/W16-2503/) and again by [Nissim et al.
(2020)](https://aclanthology.org/2020.cl-2.7/), who argue on that basis that analogy tests are a
poor instrument for measuring bias. Embeddings do encode a great deal of relational structure. They
encode less of it than the party trick implies.

**Nearest-neighbour retrieval cannot abstain, and negation barely registers.** Similarity search
always returns a top-*k*; there is no "nothing here matches", so a question your corpus cannot
answer still yields confident-looking context for the model to build on. Worse, "the drug is safe
for children" and "the drug is not safe for children" share nearly every token and are separated by
one word that the contrastive training objective was rarely forced to notice, so they sit close
together. Both problems are why a cross-encoder reranker over the top 50–100 hits is standard in
serious systems: it reads the query and passage together instead of comparing two vectors that were
computed in ignorance of each other.

**Bias is stored as geometry and travels downstream intact.** [Caliskan et al.
(2017)](https://www.science.org/doi/10.1126/science.aal4230) showed in *Science* that word
embeddings trained on web text reproduce documented human implicit-association effects — female
names sitting closer to family terms and male names closer to career terms — at effect sizes
matching the human psychology literature. Because an embedding is a substrate rather than an
endpoint, that structure is inherited by every classifier, ranker and retrieval system built on
top, which is a specific mechanism behind [algorithmic bias](https://howaiworks.ai/glossary/bias) rather than a general
worry about training data.

## Future Trends

Dimension is becoming a request parameter rather than a model property: Matryoshka training is
increasingly baked in and `dimensions` is now an argument on major embedding APIs, moving the choice
from procurement time to query time. Quantisation is heading the same way, with models trained to
survive int8 or binary storage rather than being rounded after the fact.

Multi-vector and late-interaction retrieval is moving out of research and into mainstream vector
databases, driven by exactly the capacity ceiling measured above — if one vector per document
cannot express every useful top-*k* set, more vectors per document is the structural fix, not a
bigger *D*. And instruction-conditioned models, which produce a different space depending on a task
description supplied with the input, are turning "which embedding model do I choose" into "what did
I ask it to encode" — changeable without re-selecting a model, though never without re-embedding
the corpus.

## Code Example

The worked example above, in NumPy. The point is the last four lines: once the vectors are
normalised, cosine and Euclidean distance stop being alternatives.

```python
import numpy as np

short_dog = np.array([3.0, 4.0])     # a short note about dogs
long_dog  = np.array([30.0, 40.0])   # the same content, ten times as long
database  = np.array([-4.0, 3.0])    # a note about databases

euclid = lambda a, b: np.linalg.norm(a - b)
cosine = lambda a, b: a @ b / (np.linalg.norm(a) * np.linalg.norm(b))

print(euclid(short_dog, long_dog))    # 45.0   <- "far"
print(euclid(short_dog, database))    #  7.07  <- "near", and wrong
print(cosine(short_dog, long_dog))    #  1.0
print(cosine(short_dog, database))    #  0.0

# Normalise once, and the choice of metric disappears:
unit = lambda v: v / np.linalg.norm(v)
a, b, c = unit(short_dog), unit(long_dog), unit(database)

print(euclid(a, b) ** 2, 2 - 2 * cosine(a, b))   # 0.0 0.0
print(euclid(a, c) ** 2, 2 - 2 * cosine(a, c))   # 2.0 2.0
```

Squared distance between unit vectors equals 2 − 2·cosine, exactly, so any ranking by one is a
ranking by the other. This is the identity that lets a vector index store normalised vectors and
compute a bare dot product while honestly reporting cosine similarity.

## Frequently Asked Questions

### Why use cosine similarity instead of Euclidean distance?

Because raw embedding length tracks things you usually do not want to rank on, like document length. Two documents on the same topic can sit far apart in straight-line distance simply because one is ten times longer. Cosine divides that out and compares direction only. If you normalize every vector to length 1 first, the two metrics rank identically — the squared distance between unit vectors is exactly 2 minus twice their cosine — so normalizing once at write time makes the choice disappear.

### How many dimensions do I need?

Fewer than the defaults suggest. OpenAI reported in January 2024 that text-embedding-3-large truncated to 256 dimensions scored 62.0 on MTEB, still ahead of the older text-embedding-ada-002 at 1,536 dimensions on 61.0. Dimension is not what makes an embedding good — the training objective is. Pick the smallest size that holds your retrieval quality on your own data, because storage and index memory scale linearly with it.

### Can I compare embeddings from two different models?

No. Coordinates only mean something relative to other vectors produced by the same model with the same settings; there is no shared frame between two models. Switching embedding models means recomputing every vector in your corpus, which is the real cost of the decision and the reason to test candidates before you index ten million chunks.

### What does a cosine similarity of 0.8 mean?

On its own, nothing. Embedding spaces are anisotropic — the vectors occupy a narrow cone rather than filling the space — so a model's score for two unrelated texts can sit anywhere from about 0.3 to nearly 1.0. Measure the similarity of random pairs from your own corpus first, and read every score relative to that baseline instead of treating 0.8 as 80% similar.

### Why does my retriever return an answer even when the corpus has nothing relevant?

Because nearest-neighbour search has no concept of 'no match'. It ranks everything you gave it and returns the top k, so an irrelevant chunk is always returned with some score. You need an absolute score threshold calibrated on your data, a reranker, or both — the retriever alone will never abstain.

### Do embeddings understand negation?

Usually not well. 'The drug is safe for children' and 'the drug is not safe for children' share almost every token and land close together in most embedding spaces, because the contrastive training pairs rarely forced the model to separate them. This is a live failure mode in retrieval-augmented generation and one of the strongest arguments for reranking retrieved passages with a cross-encoder.

## Related

### Related terms

- [Vector Search](https://howaiworks.ai/glossary/vector-search)
- [Semantic Search](https://howaiworks.ai/glossary/semantic-search)
- [Retrieval-Augmented Generation (RAG)](https://howaiworks.ai/glossary/retrieval-augmented-generation)
- [Dimensionality Reduction (DR)](https://howaiworks.ai/glossary/dimensionality-reduction)
- [Tokenization](https://howaiworks.ai/glossary/tokenization)
- [Representation Learning](https://howaiworks.ai/glossary/representation-learning)

---

Source: https://howaiworks.ai/glossary/embedding — HowAIWorks.ai