24 May 2026 AI Engineering6 min read

Hybrid Retrieval Is the Production Baseline

Pure vector search misses exact identifiers, error codes, and rare terms. A robust production baseline for RAG therefore combines BM25 and dense embeddings with Reciprocal Rank Fusion, then reranks the candidates with a cross-encoder. We examine accuracy gains, latency costs, and the origin of the “70% of failures” claim — and draw a clear boundary around what the approach cannot solve.

Why Pure Vector Search Fails in Production

Dense retrieval encodes queries and documents as vectors and ranks them by similarity. Its strength and weakness share the same cause: the embedding model compresses meaning into geometry. Strings with no distributional meaning — order numbers, SKUs, error codes such as ERR_CONN_RST, legal clause identifiers — have no stable place in that space. The system returns a semantically adjacent result instead of the document containing the literal token. For a support bot or contract search, adjacent is not good enough.

This weakness is documented, not anecdotal. The BEIR benchmark (Thakur et al., 2021) showed that dense retrievers trained on MS MARCO frequently failed to beat BM25 in zero-shot evaluation on unfamiliar domains. The failure is also easy to miss: evaluation sets are usually built from queries that already work, so identifier-heavy queries surface in production as hallucination complaints nobody can reproduce in the eval.

Documents are chunked, embedded and indexed — vectors plus keywords. 1/4

Two Retrievers With Complementary Failure Modes

BM25 (Robertson et al., 1994) is a lexical ranking function: it scores exact term overlap, weights terms by inverse document frequency, saturates term frequency, and normalizes for document length. It finds SKU-4471 with certainty and fails on paraphrase — a query about "terminating an agreement" never matches a document that only says "Kündigung". Dense retrieval inverts this profile: robust to synonyms and cross-lingual phrasing, blind to literal identifiers.

What matters is that the two systems fail on different queries. Elastic's hybrid-retrieval study found so little overlap between BM25 and semantic results on BEIR datasets that fusion recovered documents missed by either method alone. Hybrid retrieval is justified not by the isolated strength of one retriever, but by the independence of their failure modes.

Reciprocal Rank Fusion as the Default Merge

Reciprocal Rank Fusion (Cormack, Clarke and Büttcher, SIGIR 2009) merges ranked lists by rank position alone: score(d) = Σ 1/(k + rank_i(d)), with k = 60 by convention. Because BM25 scores are unbounded and cosine similarities live in [-1, 1], any direct score mix is miscalibrated by construction. RRF never reads the scores, so it needs no normalization, no labeled data, and no retuning when you swap the embedding model.

The measured gains are real but modest. Elastic reports that RRF adds 1.4% average NDCG@10 over its learned sparse model (ELSER) alone and 18% over BM25 alone on BEIR data. On the WANDS product-search benchmark (Turnbull, March 2025), BM25 scored 0.698 NDCG, dense 0.695, plain RRF 0.707 — and a domain-tuned hybrid 0.750, a ~7.4% lift. Read these numbers honestly: RRF's value is robustness across query types, not a large average lift. The large lifts come from the next stage.

Cross-Encoder Reranking in 2026

A cross-encoder scores each query-document pair in a single forward pass with full attention between both texts. That is more accurate than any bi-encoder similarity — and too slow for a full corpus. It therefore runs only on the top 30-100 fused candidates. The 2026 lineup is short:

The lift is measurable against the first stage: jina-reranker-v3 raises BEIR nDCG@10 from 55.81 (its own first-stage embeddings) to 61.85. Qwen3-Reranker-4B raises the MTEB English retrieval score of its 0.6B embedding stage from 61.82 to 69.76. Six to eight nDCG points is roughly what a good reranker currently buys — more than most fusion tuning ever will.

Model	Released	Access	Verified result
Cohere Rerank 4.0 (Pro / Fast)	Dec 11, 2025	API (also Azure Foundry, OCI)	32k context; Fast keeps Rerank 3.5 latency at higher accuracy (vendor claim); Pro ranked #2 in Agentset's independent ELO benchmark
jina-reranker-v3	Sep 29, 2025	Open weights + API	0.6B listwise reranker; BEIR nDCG@10 61.85 vs. 55.81 for its first-stage dense retriever
Qwen3-Reranker (0.6B / 4B / 8B)	Jun 5, 2025	Open weights (Apache 2.0)	32k context; instruction-aware; 4B scores 69.76 MTEB-R vs. 61.82 embedding-only baseline

Measured Lifts and Latency Costs

Anthropic's contextual-retrieval evaluation (September 2024) is the cleanest stacked measurement available. Baseline embeddings failed to retrieve the relevant chunk in the top 20 for 5.7% of queries. Adding contextual BM25 hybrid cut that to 2.9% (a 49% reduction); adding a reranker cut it to 1.9% — a 67% reduction in retrieval failures. A 2026 diagnostic study on the LoCoMo benchmark points the same direction: hybrid retrieval with reranking reduced retrieval-stage failures to 11.4% of questions, versus 35.3% under BM25 alone and 15.8% under cosine-only retrieval.

The cost is latency and money. Anthropic's contextual-embeddings cookbook measured roughly 100-200 ms added per query for reranking, depending on candidate-set size. Agentset's December 2025 benchmark measured Cohere Rerank 4 Fast at ~447 ms and Rerank 4 Pro at ~614 ms average per request on its harness. The reranker is the most expensive millisecond in the pipeline — which is exactly why it sees 50 candidates, not 5,000.

Where the 70 Percent Claim Comes From

The claim that "about 70% of RAG failures happen at retrieval" circulates in 70%, 72%, and 73% variants. It is usually attributed to Barnett et al., "Seven Failure Points When Engineering a RAG System" (arXiv:2401.05856, January 2024). That paper defines seven failure points — most of them upstream of generation — but provides no percentage breakdown. We could not trace the figure to any primary measurement; it is folklore wearing a citation costume.

The underlying direction nevertheless survives scrutiny. In the LoCoMo diagnostic study, retrieval failures dominated at 11-46% of all questions depending on configuration, while answer-utilization failures stayed at 4-8% and contradictions of retrieved context below 1.4%. In practice, an oracle test is enough: feed the model the gold chunk directly. If the answer becomes correct, retrieval is the bottleneck — regardless of any supposedly universal percentage.

Chunking Pragmatics

Chunking decides what a retriever can find at all. The pragmatic default is recursive splitting: divide on paragraph boundaries, fall back to sentences, then characters, until each chunk fits a target size of a few hundred tokens. It respects document structure at near-zero cost and is the default in most frameworks. Fixed-size splitting without structural awareness is strictly worse; elaborate semantic chunking, in our experience, rarely pays off before the rest of the pipeline is in place.

Contextual retrieval (Anthropic, September 2024) is the one chunking upgrade with strong published numbers: an LLM prepends a short situating description to each chunk before embedding and BM25 indexing. Contextual embeddings alone cut the top-20 retrieval failure rate by 35% (5.7% to 3.7%). The cost falls at ingestion, not at query time, and prompt caching reduces it further — a favorable trade for corpora that change slowly.

When Hybrid Is Not Enough

Hybrid retrieval fixes the ranking of evidence that already exists as a single chunk. It does not create evidence, and it does not reason across chunks. Four rules mark the boundary: 1) If the answer is not in the corpus — Barnett et al.'s first failure point — no retriever helps; detect missing content instead of ranking harder. 2) If the question aggregates ("all contracts with clause X"), top-k retrieval is the wrong primitive; you need a structured query over extracted fields.

3) If the answer requires multi-hop composition across documents that never co-rank, single-shot retrieval fails structurally. Iterative, agentic, or graph-assisted retrieval is then required; traversing typed relations between entities operates above the hybrid layer rather than replacing it. 4) If the entire corpus fits comfortably in a frontier model's context window and query volume is low, the retrieval machinery may cost more than it saves. Hybrid retrieval is the baseline, not the ceiling.

Outlook: The Stages Are Consolidating

The three-stage shape — two cheap high-recall retrievers, one fusion step, one precise ranking stage — is stable. The stages themselves are merging. jina-reranker-v3 already processes up to 64 candidate documents in a single 131k-token window, blurring the line between reranking and reading. Qwen3's instruction-aware rerankers turn ranking into a steerable step rather than a fixed function. We expect fusion and reranking to collapse into single listwise models, and retrieval itself to become a decision an agent makes repeatedly per task rather than once per query.

Two constants remain: exact identifiers still require lexical matching, and credible evaluation still requires your own labeled queries. Build the hybrid baseline first, measure the failure distribution next, and add further machinery only where the data reveals a bottleneck. That sequence shapes production retrieval quality more than any individual model choice made in 2026.