Hybrid Search: When Pure Vector Embeddings Aren't Enough

Short answer: Start dense-only. Add keyword/sparse (BM25) search when your evaluation set shows the retriever missing exact matches: product IDs, rare terms, acronyms, code symbols, and proper names. Dense embeddings capture meaning but blur these tokens, and sparse lexical search is exactly where they shine, so hybrid earns its keep only when you can measure that gap.

The failure modes of pure dense

Dense embeddings compress text into a fixed-size vector optimized for semantic similarity. That compression is the whole point, and also the problem. When the literal token is what matters, meaning-based matching underperforms. The retrieval literature is consistent here: dense retrievers miss exact and rare tokens where sparse lexical search excels.

Concretely, dense-only tends to fail on:

• Exact IDs and SKUs — ORD-99413, CVE-2024-3094. The vector for a random alphanumeric string carries almost no useful signal.
• Rare terms — long-tail vocabulary that barely appeared in the embedding model's training data.
• Acronyms — RRF, PPO, TLS. Short, overloaded, easily confused with unrelated expansions.
• Code and symbols — function names like _build_dish_filters, error strings, config keys.
• Proper names — people, drug names, part numbers, restaurant names that aren't semantically decomposable.

If your queries look like natural-language questions, dense handles them well. If they contain identifiers users copy-paste, expect misses.

Where sparse and dense complement each other

Sparse retrieval (BM25 and learned-sparse variants) scores documents on term overlap. It has no concept of synonyms or paraphrase, but it never "forgets" a rare token, because the token is an explicit dimension.

That is the exact inverse of dense's weakness. Dense nails "somewhere cozy for a rainy evening" mapping to "intimate wine bar." Sparse nails "CVE-2024-3094" landing on the one doc that contains that string. Run both and you cover paraphrase and exact match instead of trading one for the other.

How fusion works (RRF), and models that ship both

The naive merge — add the dense cosine score to the BM25 score — breaks because the two live on different, unnormalized scales. Reciprocal Rank Fusion (RRF) sidesteps this entirely: it combines the two rankings, not their scores, so no normalization is needed.

Each document gets a score summed across retrievers, where k is a small constant (typically 60) that damps the influence of low ranks:

def rrf(dense_ranking, sparse_ranking, k=60):
    scores = {}
    for ranking in (dense_ranking, sparse_ranking):
        for rank, doc_id in enumerate(ranking):  # rank starts at 0
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

A doc ranked #1 by either retriever gets a strong contribution; agreement between retrievers compounds. That's the whole trick — robust, tuning-light, and scale-agnostic.

You also don't necessarily need two models. BGE-M3 produces dense, sparse, and multi-vector (ColBERT-style) representations from a single model, so you can generate both the dense embedding and the sparse weights in one forward pass and fuse them downstream. That collapses a lot of the operational overhead people assume hybrid requires.

My take: start dense-only, add sparse when eval proves it

I default to dense-only for a new system. It's one index, one query path, one thing to reason about. Hybrid is a real jump in moving parts — a second index, sparse encoding, fusion logic, and more tuning surface.

The trigger to add sparse is not vibes, it's a labeled eval set. Build 50–200 real queries with known-correct documents, measure recall@k dense-only, then read the failures. If the misses cluster on IDs, acronyms, code, and names, add sparse and re-measure. If the misses are semantic (wrong topic entirely), sparse won't help — that's an embedding-model or chunking problem.

Cost/complexity tradeoff, and when hybrid isn't worth it

Hybrid costs you a second index to build and keep in sync, extra query latency (two retrievals plus fusion), and more parameters to tune. Skip it when:

• Queries are conversational natural language with few literal identifiers.
• Your dense eval recall is already where you need it.
• You're pre-launch with no eval set — you can't tune what you can't measure, and premature hybrid just hides the real problem.

Default to dense, instrument with an eval set, and let measured lexical misses — not hype — pull you into hybrid.

Related: more writing.