Reranking in RAG: rerankers and cross-encoders explained

Reranking is the step in a RAG pipeline that reorders the passages found by retrieval before handing them to the LLM. A specialized model, the reranker, reads each passage together with the question, gives it a relevance score, and only the best-scored passages are kept. In Anthropic's tests, adding a reranker brought the retrieval failure rate down from 2.9% to 1.9%.

Why a second step? Because RAG retrieval is fast but approximate: it compares vectors computed in advance, without ever reading the question and the passage together. The reranker does read them together. It is more accurate, but much slower: on the BEIR benchmark, reranking the top 100 results with a cross-encoder improves relevance by 11% on average, but takes 450 ms per query on a GPU, against 20 ms for keyword search alone.

So reranking has a clear limit: it sorts what retrieval found, and nothing else. In the right place, it improves answers at a modest cost. In the wrong place, it adds latency and recovers nothing. What remains is choosing between a cross-encoder, an API, an LLM judge or a plain RRF fusion, and measuring the gain on your own documents.

What does reranking do in a RAG pipeline?

A RAG system answers a question in two stages: it searches your documents for useful passages, then passes them to the LLM that writes the answer. Reranking sits between the two. Retrieval brings back a broad list of candidates, a reranker scores each of them, and the list is reordered, then cut.

Loading diagram…

The reranker does not query the index: it receives the list produced by retrieval, scores each passage against the question, and only passes the best ones on to the LLM, in order.

Volumes vary from one system to another. The Sentence Transformers documentation uses the example of around a hundred candidates to rerank. Anthropic, in its Contextual Retrieval method, reranks the top 150 passages and keeps 20.

The final order matters as much as the selection. A study published in 2023 showed that an LLM makes better use of information placed at the beginning or end of its context than in the middle. A good passage buried in twelfth position may be used less well than at the top of the list. The reranker decides what the LLM reads first.

Bi-encoder vs cross-encoder: what is the difference?

Reranking rests on two ways of comparing a question with a passage.

A bi-encoder encodes the question and each passage separately, as vectors (embeddings). Document vectors are computed once and for all, at indexing time. At query time, you only compute one vector and look for its nearest neighbors, which takes about fifteen milliseconds over a million passages on a GPU. This is classic semantic search, the kind for which we described our choice of models and providers.

A cross-encoder reads the question and the passage together, in the same model, and returns a score directly. Every word in the question can "look at" every word in the passage, which is what makes it more accurate. But nothing can be computed in advance: each question-passage pair needs a full pass through the model.

The cost gap is staggering. According to the authors of Sentence-BERT, finding the most similar pair among 10,000 sentences with a BERT cross-encoder takes about 65 hours, against 5 seconds with precomputed vectors. Hence the two-stage architecture: the bi-encoder to search, the cross-encoder to sort.

The accuracy gain has been measured since 2019. As early as January that year, a BERT-based reranker beat the previous state of the art by 27% (relative) on Microsoft's MS MARCO benchmark.

There is a middle path, late interaction, popularized by ColBERT: one vector per word rather than one per passage, compared in detail at query time. Its authors report quality competitive with BERT-based models while running two orders of magnitude faster.

The four families of rerankers

Open-weight cross-encoders

These are downloadable models that you run yourself. BAAI's (Beijing Academy of Artificial Intelligence) bge-reranker-v2-m3 has 600 million parameters, is multilingual, is released under the Apache 2.0 license, and logged more than 14 million downloads over the past month as of November 6, 2026. Alibaba publishes the Qwen3-Reranker family in 0.6, 4 and 8 billion parameters, under Apache 2.0, for more than 100 languages. Mixedbread offers mxbai-rerank-large-v2, also under Apache 2.0.

Watch the licenses: jina-reranker-v3 is under CC BY-NC 4.0, which rules out commercial use without the publisher's agreement. Downloadable weights do not guarantee free use.

The strength of this family is control. A 600-million-parameter reranker runs on your own servers, on a modest GPU, and passages from your documents never leave your infrastructure. It is the natural route for a company that requires its data to be processed in Europe.

Reranking APIs

Cohere, Voyage AI and Jina AI sell reranking per call. Cohere released Rerank 5 on October 21, 2026, priced at $0.05 per million tokens for the Pro version and $0.02 for the Fast version, and evaluated on 20 languages, French included. Voyage AI charges the same prices for rerank-3 and rerank-3-lite.

Two of these providers now belong to database and search engine vendors: MongoDB acquired Voyage AI in February 2025, and Elastic completed its acquisition of Jina AI in October 2025.

An API spares you from running a model, but every candidate passage goes to the provider. For an IT department, a reranking API is a data processor like any other, with its contract, its location and its retention policy. Cohere offers private deployment, in your own cloud or on premises, through its sales team.

The LLM as a reranker

You can also ask an LLM to judge the relevance of each passage, or to rank a list directly. Researchers showed in 2023 that ChatGPT and GPT-4, properly instructed, deliver results competitive with, or even better than, supervised methods. The same paper shows that a small 440-million-parameter model, trained to imitate those rankings, beats a 3-billion-parameter supervised model.

The advantage lies in the question you ask: with an LLM, the relevance criterion is written out in plain words instead of being frozen at training time. That is what we do with Jev, a model that answers a closed question with a probability. On its first day in production, it cut passages from 10.7 to 7.7 per search, for $0.70 per 1,000 searches.

The drawback: a general-purpose LLM costs more and responds more slowly than a specialized cross-encoder. A model designed to decide, without generating text, narrows the gap.

Hybrid search and RRF fusion

The last family is the simplest: no model at all. Hybrid search runs a keyword search and a semantic search in parallel. You then have to merge two lists whose scores cannot be compared: the lexical score has no upper bound, the semantic score is bounded.

Reciprocal Rank Fusion (RRF) gets around the problem by ignoring scores. For each list in which it appears, a document gets 1 / (60 + its rank), and the values are added up. Proposed in 2009 by researchers from the University of Waterloo and Google, it beat the best of the fused systems by 4 to 5%. Elasticsearch applies it with the same constant of 60 by default.

RRF is not a reranker in the strict sense: it does not read the text. But it is a deterministic algorithm, free and instantaneous. Before adding a model, check what RRF alone gives you.

Is reranking worth the cost?

The cost in latency

Added time is the real constraint. On BEIR, in 2021, reranking 100 passages with a small cross-encoder took 450 ms on a GPU and 6.1 seconds on a CPU, while keyword search alone took 20 ms. Hardware and models have improved: Cohere says Rerank 5 Fast processes about 187 documents of 1,000 tokens per second on an A100 card. Anthropic sums it up: reranking "inevitably adds a small amount of latency", and the more passages you rerank, the more it grows.

Two levers keep it down: fewer candidates, and parallel calls. In our platform, all reranking calls for a search go out at the same time, so the delay depends on the length of the passages, not their number.

The cost on the bill

Through an API, reranking is paid per token. Take 50 passages of 400 tokens: about 20,000 tokens per search, or $0.001 at the Cohere Rerank 5 Pro rate, or one dollar per 1,000 searches. At that scale, the reranking bill weighs little next to the bill of the LLM that writes the answer. It can even lower that bill: fewer passages sent means fewer tokens read by the LLM, as Anthropic also points out. We see the same thing with Jev: the bill is negligible, the added time is what we watch.

When to skip it

Reranking adds little or nothing in several cases:

• retrieval brings back few candidates: if you pass the top 5 passages anyway, reordering them changes little;
• the question is about an exact reference (an order number, a project code): keyword search finds the right line, there is nothing to arbitrate;
• the right passage is not in the list: a reranker finds nothing, it reorders. If retrieval misses the passage, no reranker will recover it;
• latency is critical, as in autocomplete.

This is the rule we apply everywhere: AI only where the task is not deterministic. When an algorithm is enough, it is more reliable, cheaper and lighter on resources. Well-tuned keyword search remains a serious contender: on BEIR, BM25, the classic lexical search algorithm, is "a robust baseline" that several semantic models fail to beat outside their training domain.

How do you measure what a reranker adds?

A reranker is judged on your documents and your questions, not on a public leaderboard. The method takes five steps.

1. Build a test set: 50 to 200 real questions, taken from the assistant's logs, each with the passages that answer it.
2. Measure the recall of retrieval alone: among the top 50 or 100 candidates, how many contain the right passage? That recall is the ceiling of the reranker.
3. Compare three setups: no reranker, RRF alone, and a reranker. The reference metric is nDCG@10, chosen by BEIR because it accounts for both rank and graded relevance; MRR, which looks at the rank of the first good passage, is easier to read.
4. Record latency and cost per query, looking at slow cases (95th percentile) and not just the average.
5. Read the discarded passages by hand. A reranker that cuts too much drops useful passages; it was by reading the reports, passage by passage, that we set Jev's selection rules.

One precaution: setting a score threshold (keep everything above 0.5) rather than a fixed number (keep the top 10) adapts the cut to each question. That only works if scores are comparable, which means a score normalized between 0 and 1, as bge-reranker-v2-m3 allows with a sigmoid.

What an IT department should take away

• Start with retrieval. Hybrid search and RRF first: without recall, reranking is useless.
• Measure before adding a model. A set of 50 real questions is enough to know whether the reranker gains anything on your documents.
• Pick the family based on your data. Sensitive data or a European data policy: a self-hosted open-weight cross-encoder, after checking its license. Otherwise, an API billed per token, to be treated as a data processor.
• Watch latency more than the bill. The cost of reranking is low; the added time shows up on every question.
• Write the criterion down when the business needs it. An LLM judge lets you say what counts as relevant for you ("at least partially", "document currently in force"), whereas a cross-encoder applies a notion of relevance learned elsewhere.

At Ask This Guy, reranking is part of the document search in our turnkey enterprise RAG: hybrid search, then Jev for organizations on the Worldwide AI policy. Organizations on the EU-only policy send no passages to this US provider.

Frequently asked questions about reranking and rerankers

What is a reranker?

A reranker is a model that receives a question and a list of passages already found by a search, and gives each passage a relevance score. The list is then reordered by those scores and, most of the time, cut to keep only the best. In a RAG pipeline, it sits between retrieval and the LLM that writes the answer.

What is the difference between a cross-encoder and a bi-encoder?

A bi-encoder turns the question and each document into separate vectors, computed in advance, which makes it possible to search millions of passages in milliseconds. A cross-encoder reads the question and a passage together in the same model: it is more accurate, but it has to redo the computation for every pair. So the bi-encoder is used to search and the cross-encoder to rerank a few dozen candidates.

Is reranking required in a RAG pipeline?

No. It improves relevance when retrieval brings back many candidates of uneven quality, which is common with hybrid search. It adds little if retrieval returns few passages, if the question is about an exact reference, or if the right passage was never found. The only way to decide is to measure on a set of real questions.

How many passages should you send to a reranker?

There is no universal number. Published examples range from around a hundred candidates (Sentence Transformers documentation) to 150 candidates with 20 kept (Anthropic's Contextual Retrieval). More candidates improves recall but increases latency and cost, since the reranker scores every passage. The right setting comes from measuring both.

Which reranker should you pick for non-English documents?

You need a multilingual model. Among open-weight models, bge-reranker-v2-m3 (BAAI) and Qwen3-Reranker (Alibaba) are under the Apache 2.0 license and cover many languages, French included; they can be hosted on your own servers in Europe. Among APIs, Cohere evaluates Rerank 5 on around twenty languages. In every case, test on your own documents.

Does RRF fusion replace a reranker?

Not quite. Reciprocal Rank Fusion merges several result lists (keyword and semantic) based on their ranks, without reading the text. It does not judge how relevant a passage is to the question. It is an excellent first step, free and deterministic; a reranker comes next, if measurement shows it adds something.

---

Conclusion

Reranking often improves a RAG system at little cost, as long as it stays in its place: after good retrieval, never instead of it. A cross-encoder or an LLM judge brings a fine-grained reading that vectors lack. RRF fusion already solves a good part of the problem with no model at all. Between the two, measurement decides.

To see how this reranking behaves in production, with its cost and latency figures, read our field report on Jev. And if you want to try filtered document search on your own documents, book a demo.