Reranking in RAG: Why Retrieval Quality Matters

The first retrieval stage in a RAG system must search a large collection quickly. Speed and broad recall matter, but the initial ranking is rarely perfect. [Reranking](/glossary/reranking/) adds a second, more precise relevance assessment over a smaller candidate set before context reaches the language model.

This division of labor can improve answer quality without applying an expensive model to every document. It also creates an important limit: a reranker can reorder only the candidates it receives. It cannot recover evidence that retrieval missed.

Reranking in one pipeline

A common [retrieval pipeline](/glossary/retrieval-pipeline/) has two ranking stages:

1. A fast retriever selects the top K candidates from the full index.
2. A reranker scores those candidates against the query with a more detailed method.
3. The system keeps the best few results, constructs context, and asks the LLM to answer.

The first stage seeks broad recall across the corpus. The reranker spends more compute on the small set most likely to matter.

The result is still a ranking, not a guarantee of truth. Source authority, freshness, permissions, and completeness require separate checks.

Why initial retrieval order can be weak

[Sparse retrieval](/glossary/sparse-retrieval/) scores lexical evidence such as term matches. [Dense retrieval](/glossary/dense-retrieval/) compares query and passage [embeddings](/glossary/embedding/). Both are designed to search efficiently, which limits how deeply they can model the relationship between a particular query and passage.

A dense dual-encoder usually encodes the query and documents independently. Document vectors can therefore be calculated ahead of time and searched quickly. That efficiency is powerful, but the model does not jointly examine every query word against every candidate token at retrieval time.

Initial rankings can be distorted by:

• shared keywords that occur in the wrong context;
• semantic similarity without the requested factual distinction;
• long chunks that contain several topics;
• abbreviations, numbers, and rare names;
• negation or constraints such as “not available in Europe”;
• duplicated or near-duplicated chunks;
• conversational queries that include irrelevant history.

A candidate can be plausible enough to retrieve but not precise enough to place first.

What changes after reranking

Reranking changes the order of an existing candidate set. Better scores cannot compensate for a relevant document that never entered that set.

Suppose the query asks, “Can contractors expense a standing desk?” The retriever returns:

• a general employee equipment policy;
• a contractor onboarding page;
• a contractor-specific expense exception;
• an article about ergonomic desks.

A reranker can evaluate the full query against each passage and move the contractor-specific exception above more generally similar material. The model then receives a tighter context with less distracting content.

This matters because the generation model has limited context and attention. Sending ten weak passages does not necessarily produce a safer answer than sending three strong, attributable ones.

Common reranking methods

Cross-encoder reranking

A cross-encoder processes the query and candidate text together. Joint attention lets it model fine-grained interactions that independent query and document embeddings may miss.

The cost is that each query-candidate pair needs a separate model computation. That is far too expensive for millions of documents but practical for tens or perhaps hundreds of retrieved candidates, depending on the model and latency budget.

The BERT passage reranking work by Nogueira and Cho demonstrated this pattern: retrieve candidates first, then apply a BERT-based relevance classifier.

Late-interaction models

Late-interaction approaches preserve richer token-level representations while retaining some precomputation. They can offer a balance between dual-encoder speed and cross-encoder precision, with different storage and serving trade-offs.

LLM-based reranking

An LLM can score, compare, or order candidates using explicit criteria. This can be useful for complex instructions, but it is usually more expensive and slower. Ordering bias, inconsistent outputs, long candidate text, and prompt-injection risks also require careful controls.

If an LLM reranker sees untrusted documents, treat them as data, not instructions. Use structured outputs and evaluate stability.

Rule-based and metadata-aware reranking

Not every signal needs a model. A system can boost current versions, preferred sources, exact title matches, geographic scope, or document authority. It can penalize duplicates and stale content.

These signals are valuable when they reflect a legitimate relevance or trust requirement. They should not silently override mandatory permission filters, which belong earlier in the retrieval process.

Retrieval scores and reranker scores are different

A vector similarity score describes proximity under a specific embedding and metric. A BM25 score reflects lexical evidence under its scoring formula. A cross-encoder score reflects the model and training objective used for reranking.

These values are not interchangeable and are often not calibrated probabilities. A reranker score of 0.92 does not automatically mean a passage has a 92% chance of being factually correct.

Use scores to order or threshold candidates only after analyzing their distribution on representative data. Thresholds may change when the corpus, model, or query mix changes.

Candidate depth: the key design choice

The first-stage top K controls what the reranker can see.

If K is too small, latency is low but the correct evidence may never reach the reranker. If K is too large, recall may improve while compute, latency, and noise rise. Long candidates can also exceed the reranker's input limit and be truncated.

Tune candidate depth alongside:

• sparse, dense, or [hybrid retrieval](/glossary/hybrid-search/);
• metadata and permission filtering;
• chunk size;
• reranker model;
• final number of passages;
• context construction rules.

Measure recall before reranking and ranking quality after it. Otherwise, a good final metric can hide an overly broad and costly first stage, while a bad metric may be blamed on the reranker when retrieval recall is the real problem.

Reranking vs hybrid search

Hybrid search combines lexical and semantic retrieval signals to generate or fuse candidates. Reranking applies a more precise relevance model after candidate generation.

They can be used independently or together:

• Sparse retrieval → cross-encoder reranking
• Dense retrieval → cross-encoder reranking
• Hybrid candidate union → deduplication → reranking

Hybrid search often improves coverage. Reranking often improves order. Neither claim is universal; evaluate the specific corpus.

For a direct method comparison, read [Hybrid Search vs Dense vs Sparse Retrieval](/hybrid-vs-dense-vs-sparse-retrieval/).

Reranking vs query rewriting

[Query rewriting](/glossary/query-rewriting/) changes the search expression before retrieval. Reranking assesses candidates after retrieval.

Rewriting can improve recall by adding domain vocabulary, resolving references, or separating a multi-part question. Reranking cannot expand the candidate pool, but it can distinguish candidates that the rewritten query retrieved.

An effective pipeline may use both, but each added model call increases latency and adds another failure mode. Preserve the original intent and evaluate stages independently.

How reranking affects generation

Better ordering can improve the context in several ways:

• relevant evidence reaches the limited final passage set;
• duplicates and loosely related chunks are pushed down;
• the model sees fewer conflicting topics;
• citations are attached to more directly supportive passages;
• context uses fewer tokens for the same evidentiary coverage.

Reranking does not make the generator faithful by itself. The application still needs [grounding](/glossary/grounding/) instructions, source attribution, an answerability policy, and evaluation of whether claims are supported.

If retrieved sources conflict, the system may need to prefer an authoritative source, show the disagreement, or ask for clarification. Pure relevance ranking may put a persuasive but outdated passage first.

When reranking is worth it

Reranking is especially useful when the first-stage retriever often finds the correct passage but ranks it below distracting candidates. It can also help when a large corpus has many semantically similar policies, products, or procedures.

It may be unnecessary when the corpus is tiny, exact filters already isolate a single record, the top result is consistently sufficient, or the latency budget is extremely tight. Start by inspecting failures. Do not add a reranker only because the architecture diagram looks more complete.

How to evaluate a reranker

Build a test set with queries and graded relevance judgments. Include exact identifiers, paraphrases, negatives, current-versus-old versions, permission scopes, and queries with no supported answer.

Evaluate at least three layers:

Candidate recall

Does first-stage retrieval include relevant evidence within top K? If not, fix retrieval, indexing, or query processing.

Ranking quality

Does the reranker move stronger evidence upward? Metrics such as mean reciprocal rank or normalized discounted cumulative gain can summarize ordering, while manual inspection explains why it changed.

Downstream quality

Do generated answers become more correct, grounded, and citation-aligned? What happens to latency, cost, and token use? Retrieval metrics are a proxy, not the final user outcome.

Run ablations: the same pipeline with and without reranking. Test a smaller candidate set and simpler model. Segment results by query type. An average gain can conceal regressions on IDs, negations, or another important class.

Common mistakes

Reranking too few candidates

The model looks precise, but relevant evidence is absent. Increase or diversify first-stage retrieval only if evaluation shows a recall problem.

Sending every reranked candidate to the LLM

Reranking should help select context, not justify unlimited context. Use a final cutoff based on relevance, diversity, and token budget.

Ignoring duplicates

Near-duplicate chunks can dominate the top positions. Deduplicate or diversify so the context represents independent useful evidence.

Treating relevance as authority

A highly relevant unofficial note may rank above the current policy. Add trusted metadata and source-priority rules where the domain requires them.

Optimizing only an offline ranking metric

A metric gain that adds unacceptable latency or does not improve grounded answers may not be a product improvement.

Practical implementation sequence

Begin with a strong retriever and labeled test queries. Record its candidate recall and final answer quality. Add a reranker over a modest top K, keep only the best few diverse results, and compare the full system against the baseline.

Instrument each stage so a failed answer can be traced to the query, retrieved candidates, reranker order, assembled context, and final response. That makes it possible to distinguish “the evidence was never found” from “the evidence was found and then mishandled.”

Reranking earns its place when it consistently converts retrieved candidates into better evidence for the model at an acceptable cost.

Sources

• Passage Re-ranking with BERT describes a BERT-based second-stage relevance classifier for passage ranking.
• Dense Passage Retrieval for Open-Domain Question Answering presents an efficient dual-encoder retrieval approach and provides context for the first-stage/second-stage distinction.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks describes a foundational retrieval-plus-generation architecture.

Continue learning

Read [How RAG Works](/how-rag-works/) for the complete pipeline, then use [Build Your First RAG Agent](/build-your-first-rag-agent/) to apply retrieval and ranking inside a goal-directed system.