Key takeaway: Dense and sparse retrieval fail in complementary ways. Combining them, then reranking the merged set, outperforms either approach used alone.
Where Each Method Breaks
Vector search finds documents whose meaning resembles the query. A search for “how do I stop the service” retrieves a document about shutting down a process even with no shared words.
That same mechanism handles exact matches badly. A query for part number XR-4471-B becomes a vector near other alphanumeric strings, and the specific document may not appear at all. Rare technical terms, proper nouns, version numbers and error codes all suffer the same way, because the model compresses them into an approximate region rather than treating them as exact tokens.
Keyword search — BM25 and its relatives — is exact by construction. It finds XR-4471-B immediately and returns nothing for a paraphrase that shares no vocabulary.
| Query type | Dense | Sparse |
|---|---|---|
| Conceptual question | Strong | Weak |
| Paraphrased request | Strong | Weak |
| Exact identifier | Weak | Strong |
| Rare technical term | Weak | Strong |
| Quoted phrase | Weak | Strong |
| Multi-concept query | Moderate | Weak |
Merging Results Properly
Running both retrievers produces two ranked lists with incomparable scores — a cosine similarity of 0.82 and a BM25 score of 14.3 have no common scale.
Normalising the scores to combine them is fragile, because score distributions shift with query and corpus. Reciprocal rank fusion avoids the problem entirely by using position rather than score: each document’s contribution is a function of its rank in each list, and the fused ranking needs no calibration.
That property makes it robust to retriever changes. Swapping the embedding model does not require retuning the fusion, because ranks remain ranks.
Reranking as the Final Stage
Retrieval optimises for recall — getting the right document into the candidate set. Precision at the top is a separate problem that a reranker solves.
A cross-encoder reranker takes the query and each candidate together and scores their relevance directly. Because it sees both texts jointly rather than comparing independent vectors, it is substantially more accurate.
It is also far too slow to run over a whole corpus, which is why the architecture is two-stage: retrieve fifty candidates cheaply, rerank those fifty expensively, return the top five. The reranker’s cost is bounded by the candidate count rather than the corpus size.
This stage typically produces the largest single quality improvement in a retrieval pipeline, and it is frequently the last thing teams add.
Practical Construction
Retrieve twenty to fifty candidates from each retriever, fuse by reciprocal rank, rerank the merged set, and pass the top few to generation.
Weight the fusion toward whichever retriever suits your content. A corpus of technical documentation full of identifiers benefits from a heavier sparse weight; conversational content benefits from the reverse.
Measure each stage separately. Recall at fifty tells you whether retrieval is finding the answer at all; precision at five tells you whether reranking is surfacing it. A single end-to-end number cannot distinguish a retrieval failure from a ranking failure, and the fixes are different.
The Bottom Line
Run dense and sparse retrieval together, fuse by reciprocal rank rather than normalised scores, then rerank the merged candidates with a cross-encoder. Measure recall and precision at separate stages so you know which one to improve.
