Hybrid search for RAG: combining BM25 and dense vectors with RRF

Short answer: run BM25 (sparse) and embedding (dense) retrieval in parallel, then combine their ranked lists with Reciprocal Rank Fusion (RRF). Don't average the raw scores — fuse on rank position. Expect an 8–15% accuracy gain for almost no extra latency.

Dense vector search is great at meaning but weak at exact matches — product IDs, error codes, acronyms, rare names. Keyword search (BM25) is the opposite. Hybrid gives you both.

The architecture

Two retrievers, run at the same time, then merged:

• Sparse (BM25 / miniCOIL): nails exact terms, identifiers, acronyms.
• Dense (embeddings): nails paraphrase and semantic similarity.
• Fusion (RRF): merges the two ranked lists into one.

Why you can't just average the scores

The classic mistake is to blend the two by adding or averaging their scores. It doesn't work because the scales are incompatible:

• BM25 is unbounded and positive (0 → large).
• Cosine similarity is roughly −1 to 1.

Average them and BM25's magnitude quietly dominates. So instead of fusing scores, you fuse ranks.

Reciprocal Rank Fusion in one formula

For each document, sum 1 / (k + rank) across every result list it appears in (a common default is k = 60):

def rrf(result_lists, k=60):
    scores = {}
    for results in result_lists:              # e.g. [bm25_hits, dense_hits]
        for rank, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

A document ranked #1 by BM25 and #3 by dense search rises to the top; one that only appears deep in a single list stays low. No score-scale problems, because only positions matter.

Practical settings

• Candidate depth: pull a generous top-k from each retriever (top-50 to top-500 depending on corpus), fuse, then trim.
• Add a reranker after fusion when you need top-tier precision — retrieve wide, fuse, rerank to the final 3–5. (See the reranker guide.)
• Run the two retrievals concurrently so hybrid costs roughly the latency of one call.

When hybrid matters most

If your queries are natural-language and fuzzy, pure dense is often fine. Hybrid earns its keep when users search for rare terms, exact identifiers, acronyms, or code — exactly where embeddings alone quietly fail.

Sparse + dense in parallel, fused on ranks with RRF, optionally reranked. It's the highest-ROI upgrade after chunking.

Related: how to choose chunk size, and why I don't use LangChain in production.