Reranking (Cross-Encoder)

Definition

Reranking is the second stage of a two-stage search pipeline. A fast first-stage retriever pulls a few dozen to a few hundred candidate documents out of a large collection, and a slower, more accurate model then rescores only those candidates and reorders them. In modern systems that second-stage model is almost always a cross-encoder: a transformer that reads the query and one candidate document concatenated together, in a single forward pass, and emits one number — how relevant this document is to this query.

The reason the pipeline is split in two, rather than using the accurate model for everything, is a hard cost asymmetry. A cross-encoder is more accurate precisely because it looks at the query and the document together — and that is exactly what stops its scores from ever being precomputed, so it has to run once per candidate at query time. Reranking is the engineering compromise that buys the cross-encoder's precision without paying to run it over the whole corpus. It is the difference, in a retrieval-augmented generation system, between the language model reading the passage that actually answers the question and reading the fifth-most-similar paragraph that happens to share some vocabulary with it.

How It Works

To see why two stages beat one, you have to look at what the two kinds of encoder do differently with the same transformer.

A bi-encoder — the model behind ordinary vector search — runs the query and each document through the network separately. Each becomes a single fixed embedding, and relevance is just the dot product or cosine similarity between the two vectors. The crucial property is that the document is encoded with no knowledge of the query: its vector is the same no matter who searches for it. So you encode every document in the collection once, offline, store the vectors in a vector database, and at query time you encode only the query and look up its nearest neighbours. The work at query time is fixed and tiny — one forward pass for the query plus an approximate-nearest-neighbour lookup that touches a small fraction of the index — no matter whether the collection holds ten thousand documents or ten million.

A cross-encoder gives up exactly that reusability to gain accuracy. It concatenates the query and one document into a single input — [CLS] query [SEP] document [SEP] — and runs the whole thing through the transformer at once. Now the model's self-attention lets every query token attend to every document token and vice versa. The model can notice that "not safe for children" and "safe for children" are opposites rather than near-duplicates; it can see that a pronoun in the document refers to the entity the query asked about; it can weigh a document's third sentence against the query's second clause. A bi-encoder cannot do any of this, because when it encoded the document it had never seen the query. This joint reading is why cross-encoders top the accuracy tables — and why their scores cannot be cached. The document has no fixed representation to store; its "meaning" is different for every query, and there is nothing to compute until the query arrives.

That single design choice sets the whole pipeline. Put the two kinds of interaction on a spectrum and the trade-off is visible end to end: no interaction (the bi-encoder — fully precomputable, scales to billions, least accurate), late interaction (ColBERT-style models that store one vector per token and compare them at query time — partly precomputable, a middle ground), and full interaction (the cross-encoder — nothing precomputable, most accurate). Reranking is how you get the accuracy of the expensive end of that spectrum while paying, most of the time, for the cheap end.

The cost that forces two stages

The arithmetic is what makes this concrete. A small BERT-class cross-encoder scores a single query-document pair on the order of a few milliseconds on a GPU — the 2021 BEIR study measured about 350 ms to rerank 100 candidates, which is roughly 3.5 ms per pair. Hold that per-pair cost fixed and ask what it buys at different scales.

Rerank a shortlist of 100 candidates: 100 pairs times 3.5 ms is about 350 ms — a noticeable but acceptable slice of a search response.

Now score an entire corpus of 10 million documents the same way: 10,000,000 pairs times 3.5 ms is 35,000 seconds, or nearly 10 hours for one query. That is not a slow system; it is not a system at all.

The ratio between those two is the whole argument, and it does not depend on the per-pair time you assume: reranking a shortlist of N candidates instead of the full corpus of M documents is exactly M / N times cheaper — here 10,000,000 / 100 = 100,000× cheaper. The first stage's job is to produce that shortlist cheaply enough that the ratio is affordable, and to make it good enough that the answer is somewhere inside it. A bi-encoder does this because its query-time cost is a constant handful of forward passes regardless of corpus size, where the cross-encoder's cost grows linearly with the number of documents it has to score. Two stages exist to spend the constant-cost model on recall over everything and the linear-cost model on precision over a shortlist — and never the other way around.

Real-World Applications

Retrieval-augmented generation. The retrieve-then-rerank pattern is close to universal in RAG. A document is split into chunks, each chunk is embedded and stored in a vector database, and a query embedding pulls the top 50 to 100 chunks by vector similarity. Those candidates then go through a cross-encoder, and only the top handful survive into the language model's context window. The division of labour is deliberate: embedding search is tuned for recall (cast a wide net so the answer is somewhere in the net), and the reranker is tuned for precision (put the answer at the top). Skipping the reranker is one of the most common reasons a RAG system retrieves plausibly-related but subtly-wrong passages — the failure that embedding negation ("safe" versus "not safe") produces and that a cross-encoder is specifically good at catching.

Open-source rerankers. The sentence-transformers library ships a CrossEncoder class and a family of models under the cross-encoder/ms-marco-* names — small MiniLM-based models trained on the MS MARCO passage-ranking dataset, which is a large collection of real Bing queries paired with human relevance judgements. These are the default drop-in reranker for a great many production stacks, precisely because they are small enough that a few hundred pairs fit comfortably in a query's latency budget.

Reranking as a hosted service. Cohere offers a Rerank API — you send a query and a list of candidate documents and get back relevance scores — so teams can add a reranking stage without training or serving a model themselves. Search platforms have moved the same way: Elasticsearch and OpenSearch added semantic reranking features that call a cross-encoder over the top hits from a first-stage BM25 or vector query, which is the two-stage pipeline packaged as a single search request.

Benchmark evidence that it is worth it. The BEIR generalization study (Thakur et al., NeurIPS 2021) evaluated ten retrieval systems across 18 datasets and reported, in its own words, that "re-ranking and late-interaction-based models on average achieve the best zero-shot performances, however, at high computational costs." Concretely, a BM25 first stage followed by a cross-encoder reranker over the top 100 hits beat plain BM25 on 16 of the 18 datasets and improved average nDCG@10 by about 11%, more than any single-stage neural retriever the study tested. The figures are from 2021 and the specific models have been superseded, but the structural result — accuracy sits at the expensive, jointly-encoded end, and you reach it through a shortlist — has not changed.

Key Concepts

• First-stage recall is the ceiling. A reranker can only reorder the candidates it is handed. If the answer is not in the top-N the first stage returned, no reranker in the world recovers it — it is not in the pile to be reordered. This is why the first stage is tuned for recall, why N is set generously, and why production systems often run two first-stage retrievers (BM25 for exact terms, a dense retriever for meaning) and merge their candidate lists: two uncorrelated recall failures are safer than one.
• N is the latency dial. Every extra candidate is one more forward pass, so reranking cost is linear in N while recall improves with sharply diminishing returns. Moving from the top 50 to the top 100 might rescue a few borderline answers; moving to the top 1,000 rarely earns its ten-fold latency cost. You set N where your recall need meets your latency budget.
• Scores rank but do not calibrate. A cross-encoder's output orders documents within one query well. It is not a probability and it is generally not comparable across queries, so you cannot compare one query's top score against another's to decide whether either found anything good. Answering "is anything relevant at all?" requires calibrating a threshold on your own data.
• The teacher can train the student. Because the cross-encoder is the accurate one, it is often used as a teacher: its scores become training targets for the cheaper bi-encoder (as in margin-MSE distillation). This pushes some of the cross-encoder's judgement into the first stage, which raises first-stage recall and lets you rerank fewer candidates for the same quality.

Challenges

The recall ceiling fails silently. The most dangerous property of reranking is that a broken first stage looks fine through the reranker. On every query where retrieval worked, the reranker does its job and results look excellent; on the queries where the answer never made the shortlist, the reranker cannot help and there is no error — just a confidently-ranked list of wrong-but-related documents. The only way to see this is to measure the two stages separately: recall@N on the first stage, ranking quality on the second. A team that measures only final answer quality will tune the reranker for weeks when the first stage was the problem.

Latency under real load. The comfortable few-hundred-milliseconds figure is for one query in isolation. On a server already saturated with concurrent requests, every query's 100 forward passes compete with every other query's for the same GPU. Batching pairs together raises throughput, but it does nothing for tail latency, and a reranker is often the single most expensive component in a search request's budget. Sizing N is as much a capacity-planning decision as a quality one.

Domain mismatch in the training data. The popular open cross-encoders are trained on MS MARCO — web search questions and answers. Point one at legal contracts, clinical notes, or code and it can underperform a much simpler baseline, because "relevance" in those domains does not look like relevance on the open web. Reranking is not free accuracy; a reranker trained on the wrong distribution can rank worse than the first stage it was meant to improve, and the only way to know is to evaluate on your own labelled queries.

The temptation to over-rerank. Because more candidates weakly improves recall, there is constant pressure to raise N "just to be safe," and because scores look like confidence there is pressure to trust them as thresholds. Both are traps: the first spends latency on candidates the first stage already ranked hopeless, and the second reads calibration into a number that has none.

Future Trends

Large language models as rerankers. Instead of a small cross-encoder scoring one pair at a time, you can hand a capable LLM the query and all N candidates at once and ask it to rank them — a listwise reranker, of which RankGPT-style prompting is the best-known example. Reading all candidates together lets the model reason about them relatively rather than in isolation, and it tends to produce the highest-quality ordering available. It is also far slower and more expensive per query, so it is showing up where the top few results matter enormously and the query volume is low, rather than as a default.

Pushing accuracy down into the first stage. Cross-encoder distillation and late-interaction models (the ColBERT line) both aim at the same target: recover most of the cross-encoder's accuracy without its query-time cost, so the reranker has less work left to do — or, in the limit, so a shortlist is barely needed at all. As these first stages improve, the reranker reranks fewer candidates for the same final quality, and the 100,000× gap between the two stages narrows from the retrieval side.

Code Example

A reranking stage is short — the first stage has already done the hard narrowing. This uses sentence-transformers to rescore a shortlist and keep the best five:

from sentence_transformers import CrossEncoder

# A small cross-encoder trained on MS MARCO relevance pairs.
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

query = "How do I cancel my subscription?"

# The candidates a fast first stage (BM25 or a vector search) already
# retrieved. In production this list has ~100 entries, not three.
candidates = [
    "To cancel, open Settings, then Billing, and click End plan.",
    "Our subscription plans start at nine dollars per month.",
    "Password reset links expire after 24 hours for security.",
]

# Score the query JOINTLY against each candidate: one forward pass per
# pair. This is the expensive step, which is why the list stays short.
pairs = [(query, doc) for doc in candidates]
scores = reranker.predict(pairs)

# Sort by relevance and keep the top results to pass to the LLM.
ranked = sorted(zip(scores, candidates), reverse=True)
for score, doc in ranked[:5]:
    print(round(float(score), 2), doc)

Three things in this snippet carry the whole idea. First, predict runs the transformer once for every (query, doc) tuple — the cost is linear in the number of candidates, which is why the input is a shortlist and not the corpus. Second, the billing-cancellation candidate wins even though it shares few words with the query, because the cross-encoder reads the query and document together and understands what "cancel my subscription" is asking for — a plain keyword or embedding match would rank the "plans start at" line highly on surface similarity. Third, the returned scores are raw relevance values: fine for the sorted call that orders this query's candidates, but not probabilities and not safe to compare against another query's scores.