07 Dec 2023 AI Engineering5 min read

Hybrid Search: BM25 Is Not Dead

Pure vector search misses part numbers, error codes, and other exact identifiers. We explain the complementary strengths of BM25 and dense retrieval, show how Reciprocal Rank Fusion merges them without score calibration, and assess the measured gains from hybrid search and early rerankers such as Cohere Rerank using 2023 benchmarks.

Why pure vector search misses exact identifiers

In 2023, one stack became the default for RAG applications: split documents into chunks, embed each chunk with a model such as OpenAI's text-embedding-ada-002, store the vectors in an ANN index, and retrieve by cosine similarity. The approach handles paraphrased questions well. It is predictably weak, however, on everyday exact queries for part numbers, error codes, invoice IDs, function names, or references to standards.

The failure is structural, not a tuning problem. An embedding compresses a chunk into a vector of fixed length — 1,536 dimensions for ada-002. Rare identifiers such as DIN 4108-2 or a specific SKU contribute almost nothing to that vector. Microsoft quantified this in September 2023: on keyword-style queries, pure vector retrieval scored 11.7 NDCG@3 while plain keyword search scored 79.2. The exact match the user asked for is simply not represented.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

What BM25 still does better

BM25 is a ranking function from the Okapi system, first evaluated at TREC-3 in 1994. It scores a document by term frequency, inverse document frequency and document length, controlled by two parameters, k1 and b. It matches exact tokens, its scores are explainable, it needs neither training data nor a GPU, and the inverted index is cheap to build and to update.

It is also hard to beat out of domain. The BEIR benchmark (Thakur et al., 2021) compared retrieval systems zero-shot across 18 datasets and found BM25 to be a strong baseline that many dense retrievers trained on MS MARCO failed to beat on unfamiliar corpora. What BM25 does not do: it has no notion of synonyms or paraphrase. "Kündigungsfrist" and "notice period" share no tokens. That gap is real — and it is exactly the gap embeddings close.

What dense embeddings add

Dense retrieval maps queries and documents into the same vector space with a bi-encoder. Relevance becomes geometric proximity. This closes the vocabulary gap: paraphrases, synonyms and — with multilingual models — queries across language boundaries find the right passages without shared tokens. Since December 2022, text-embedding-ada-002 has been the pragmatic default; in 2023 open models such as E5, GTE and BGE reached comparable quality on the MTEB benchmark.

The limits are equally concrete: quality falls on domains far from the training data, cosine scores have no calibrated meaning, and exact identifiers remain a blind spot because the encoder was never optimized to preserve them. Dense retrieval closes the semantic gap left by lexical search, but cannot replace its exact matches.

Reciprocal Rank Fusion in one formula

Reciprocal Rank Fusion was published by Cormack, Clarke and Büttcher at SIGIR 2009 as a two-page paper. The rule: RRF-score(d) = Σ 1/(k + r(d)), summed over all result lists, where r(d) is the document's rank in a list and k = 60. In their TREC experiments this simple rule beat Condorcet Fuse and the best individual system by 4 to 5 percent.

The decisive property: RRF consumes ranks, not scores. BM25 scores are unbounded; cosine similarities live in [-1, 1]. RRF never has to normalize either. It requires no training and no tuning; the paper found k = 60 near-optimal but not critical. By late 2023 it is productized: Elasticsearch 8.8 (May 2023) ships RRF in the search API, Weaviate has offered hybrid queries since v1.17 (December 2022), and Azure Cognitive Search fuses keyword and vector results with RRF.

That simplicity has a price: RRF ignores score magnitudes, so a decisive top result and a marginal one count equally at the same rank. A weak retriever can also dilute a strong one in the mix. Fusion-window size and any per-list weighting therefore remain deliberate design choices.

Measured gains from hybrid retrieval

In September 2023, Microsoft published the year's most cited hybrid-search figures, measured on Azure Cognitive Search with ada-002 vectors, 512-token chunks, and RRF fusion. The pattern matters more than the absolute values: every stage improves relevance, and each does so for a different reason.

Hybrid beats both single-method configurations on every benchmark in the study. The reranker delivers the largest single step on customer data. One caveat is due: this is a vendor benchmark using the vendor's own reranker. But the direction matches what we at Blue IT Systems measure in client projects: fusing lexical and dense retrieval is the cheapest relevance win available — it costs one extra query against an index you probably already operate.

Configuration	Customer datasets (NDCG@3)	BEIR (NDCG@10)
Keyword (BM25)	40.6	40.6
Vector (ada-002)	43.8	45.0
Hybrid (RRF)	48.4	48.4
Hybrid + semantic reranker	60.1	50.0

Rerankers as a second stage

A reranker is a cross-encoder: it reads query and candidate document together and outputs one relevance score. That joint attention is more accurate than any bi-encoder comparison — and far too slow to run over a full corpus. Hence the two-stage pattern: retrieval (BM25, dense or hybrid) selects 50 to 100 candidates; the reranker reorders them.

In 2023, this pattern became an API call. Cohere launched Rerank on May 1 with rerank-english-v2.0 and rerank-multilingual-v2.0. In its own evaluation, lexical search placed a relevant result in the top 3 for about 44% of queries, embedding search for 65%, and reranking for 72%. The exchange is explicit: additional latency, per-call cost, and a hard recall ceiling. No reranker can recover a document that the first stage never retrieved.

Outlook from December 2023

Writing in December 2023, we expect three developments. First, hybrid retrieval moves from expert feature to default; search engines already ship fusion as a one-line query parameter. Second, learned sparse models such as SPLADE and Elastic's ELSER blur the boundary — they produce term weights like BM25 but learn expansion like an embedding model. Third, rerankers get smaller and cheaper and will increasingly run self-hosted next to the index.

Our prediction is therefore that BM25 will still sit beside the current embedding model in production retrieval pipelines five years from now. Fusing heterogeneous rankings is a more durable idea than any individual model. If you build RAG today, keep the inverted index and measure every stage on your own queries rather than a vendor's benchmark.